Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 02:56:59.041611 21350 test_util.cc:274] Using random seed: 2067740360
2025-01-12T02:56:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T02:56:59Z Disabled control of system clock
I20250112 02:56:59.105388 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:34893
--webserver_interface=127.20.217.190
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 02:56:59.384613 21359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:56:59.385177 21359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:56:59.385637 21359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:56:59.414778 21359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:56:59.415071 21359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:56:59.415308 21359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:56:59.415539 21359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:56:59.448284 21359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.217.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:56:59.449646 21359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:56:59.451144 21359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:56:59.464682 21365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:56:59.465035 21366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:56:59.467396 21359 server_base.cc:1034] running on GCE node
W20250112 02:56:59.466087 21368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:00.604826 21359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:00.607852 21359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:00.609253 21359 hybrid_clock.cc:648] HybridClock initialized: now 1736650620609186 us; error 71 us; skew 500 ppm
I20250112 02:57:00.610265 21359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:00.621956 21359 webserver.cc:458] Webserver started at http://127.20.217.190:39919/ using document root <none> and password file <none>
I20250112 02:57:00.622915 21359 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:00.623116 21359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:00.623503 21359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:00.627964 21359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data/instance:
uuid: "bcc0757376c3411a9fcbc84ec411befd"
format_stamp: "Formatted at 2025-01-12 02:57:00 on dist-test-slave-gppw"
I20250112 02:57:00.629031 21359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal/instance:
uuid: "bcc0757376c3411a9fcbc84ec411befd"
format_stamp: "Formatted at 2025-01-12 02:57:00 on dist-test-slave-gppw"
I20250112 02:57:00.635496 21359 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250112 02:57:00.640391 21375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:00.641281 21359 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250112 02:57:00.641546 21359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal
uuid: "bcc0757376c3411a9fcbc84ec411befd"
format_stamp: "Formatted at 2025-01-12 02:57:00 on dist-test-slave-gppw"
I20250112 02:57:00.641837 21359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:00.688163 21359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:00.689550 21359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:00.689985 21359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:00.754125 21359 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.190:34893
I20250112 02:57:00.754192 21426 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.190:34893 every 8 connection(s)
I20250112 02:57:00.756597 21359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
I20250112 02:57:00.761446 21427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:00.761682 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21359
I20250112 02:57:00.762053 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-0/wal/instance
I20250112 02:57:00.765424 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:36049
--webserver_interface=127.20.217.189
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:57:00.772142 21427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:00.794857 21427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:57:00.797101 21379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
W20250112 02:57:00.800715 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:00.850154 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 1
W20250112 02:57:00.853761 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:00.922112 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 2
W20250112 02:57:00.925768 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:01.017117 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 3
W20250112 02:57:01.020767 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
W20250112 02:57:01.054749 21429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:01.055308 21429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:01.055753 21429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:01.085533 21429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:57:01.085861 21429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:01.086110 21429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:57:01.086344 21429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:57:01.119808 21429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:36049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.217.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:01.121094 21429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:01.122594 21429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:01.136549 21439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:01.164101 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 4
W20250112 02:57:01.168884 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:01.468226 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 5
W20250112 02:57:01.472941 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:02.020344 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 6
W20250112 02:57:02.029129 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:36049: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
W20250112 02:57:01.137245 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:01.138234 21429 server_base.cc:1034] running on GCE node
W20250112 02:57:01.139267 21441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:02.301891 21429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:02.304327 21429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:02.305634 21429 hybrid_clock.cc:648] HybridClock initialized: now 1736650622305603 us; error 52 us; skew 500 ppm
I20250112 02:57:02.306349 21429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:02.312314 21429 webserver.cc:458] Webserver started at http://127.20.217.189:45653/ using document root <none> and password file <none>
I20250112 02:57:02.313185 21429 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:02.313365 21429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:02.313799 21429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:02.318356 21429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/instance:
uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
format_stamp: "Formatted at 2025-01-12 02:57:02 on dist-test-slave-gppw"
I20250112 02:57:02.319439 21429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal/instance:
uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
format_stamp: "Formatted at 2025-01-12 02:57:02 on dist-test-slave-gppw"
I20250112 02:57:02.326117 21429 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250112 02:57:02.331224 21450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:02.332172 21429 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250112 02:57:02.332461 21429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
format_stamp: "Formatted at 2025-01-12 02:57:02 on dist-test-slave-gppw"
I20250112 02:57:02.332772 21429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:02.377910 21429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:02.379371 21429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:02.379784 21429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:02.444718 21429 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.189:36049
I20250112 02:57:02.444806 21501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.189:36049 every 8 connection(s)
I20250112 02:57:02.447296 21429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
I20250112 02:57:02.452157 21502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:02.454499 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21429
I20250112 02:57:02.454981 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal/instance
I20250112 02:57:02.458132 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.188:35683
--webserver_interface=127.20.217.188
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:57:02.467013 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:02.486749 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:02.499963 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:57:02.502100 21453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
W20250112 02:57:02.505903 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:02.555478 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 1
W20250112 02:57:02.559180 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:02.627655 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 2
W20250112 02:57:02.631333 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:02.722733 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 3
W20250112 02:57:02.726616 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
W20250112 02:57:02.753543 21504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:02.754089 21504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:02.754539 21504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:02.783993 21504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:57:02.784282 21504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:02.784524 21504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:57:02.784754 21504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:57:02.816674 21504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.188:35683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.217.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:02.817981 21504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:02.819494 21504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:02.833222 21515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:02.870095 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 4
W20250112 02:57:02.874229 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.089807 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } attempt: 7
I20250112 02:57:03.100579 21427 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:57:03.106437 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.164809 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 1
W20250112 02:57:03.168383 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.173599 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 5
W20250112 02:57:03.177415 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.250025 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 2
W20250112 02:57:03.253954 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.339496 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 3
W20250112 02:57:03.343493 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.484177 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 4
W20250112 02:57:03.487574 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.724773 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 6
W20250112 02:57:03.732180 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
I20250112 02:57:03.770952 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 5
W20250112 02:57:03.774574 21427 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:35683: Network error: Client connection negotiation failed: client connection to 127.20.217.188:35683: connect: Connection refused (error 111)
W20250112 02:57:02.834314 21517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:02.835184 21504 server_base.cc:1034] running on GCE node
W20250112 02:57:02.833750 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:04.009025 21504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:04.011525 21504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:04.012850 21504 hybrid_clock.cc:648] HybridClock initialized: now 1736650624012828 us; error 44 us; skew 500 ppm
I20250112 02:57:04.013739 21504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:04.020217 21504 webserver.cc:458] Webserver started at http://127.20.217.188:33195/ using document root <none> and password file <none>
I20250112 02:57:04.021060 21504 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:04.021267 21504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:04.021693 21504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:04.026434 21504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data/instance:
uuid: "e76221f7d4ec44da987303d68c393a1b"
format_stamp: "Formatted at 2025-01-12 02:57:04 on dist-test-slave-gppw"
I20250112 02:57:04.027547 21504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal/instance:
uuid: "e76221f7d4ec44da987303d68c393a1b"
format_stamp: "Formatted at 2025-01-12 02:57:04 on dist-test-slave-gppw"
I20250112 02:57:04.034623 21504 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250112 02:57:04.039873 21529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:04.040802 21504 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250112 02:57:04.041090 21504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal
uuid: "e76221f7d4ec44da987303d68c393a1b"
format_stamp: "Formatted at 2025-01-12 02:57:04 on dist-test-slave-gppw"
I20250112 02:57:04.041385 21504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:04.087559 21504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:04.088932 21504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:04.089327 21504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:04.155267 21504 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.188:35683
I20250112 02:57:04.155324 21580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.188:35683 every 8 connection(s)
I20250112 02:57:04.157833 21504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
I20250112 02:57:04.161598 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21504
I20250112 02:57:04.162034 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-2/wal/instance
I20250112 02:57:04.163513 21581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:04.180133 21581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:04.204093 21581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:04.215130 21581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:04.232837 21581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Bootstrap starting.
I20250112 02:57:04.238444 21581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:04.240485 21581 log.cc:826] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:04.244458 21581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: No bootstrap required, opened a new log
I20250112 02:57:04.265082 21581 raft_consensus.cc:357] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.265728 21581 raft_consensus.cc:383] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:04.265987 21581 raft_consensus.cc:738] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Initialized, Role: FOLLOWER
I20250112 02:57:04.266748 21581 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.269424 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.270010 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:04.270818 21581 sys_catalog.cc:564] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:57:04.283151 21592 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:04.283789 21592 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.287608 21592 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
W20250112 02:57:04.306336 21603 catalog_manager.cc:1559] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:57:04.306674 21603 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:57:04.327150 21427 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 6
I20250112 02:57:04.291152 21504 master_runner.cc:186] Error getting master registration for 127.20.217.190:34893: OK, instance_id { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" instance_seqno: 1736650620735978 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:57:04.332190 21504 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:57:04.343866 21535 tablet.cc:2367] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:57:04.357774 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Bootstrap starting.
I20250112 02:57:04.365309 21427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:04.367689 21427 log.cc:826] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:04.372841 21427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: No bootstrap required, opened a new log
I20250112 02:57:04.403026 21427 raft_consensus.cc:357] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.403894 21427 raft_consensus.cc:383] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:04.404171 21427 raft_consensus.cc:738] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc0757376c3411a9fcbc84ec411befd, State: Initialized, Role: FOLLOWER
I20250112 02:57:04.404989 21427 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.407918 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.408763 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:04.409864 21427 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:57:04.410181 21402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:04.411572 21402 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 0.
I20250112 02:57:04.413478 21531 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters:
I20250112 02:57:04.414556 21592 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:04.414994 21592 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:04.415413 21592 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:04.427667 21592 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.439658 21402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd"
I20250112 02:57:04.440215 21402 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:04.440654 21592 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 1 election: Requested vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
W20250112 02:57:04.450853 21381 tablet.cc:2367] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:57:04.450897 21402 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 1.
I20250112 02:57:04.452171 21531 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters:
I20250112 02:57:04.453038 21592 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:04.473487 21592 raft_consensus.cc:695] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 LEADER]: Becoming Leader. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Running, Role: LEADER
I20250112 02:57:04.474558 21592 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:04.491458 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.492271 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
W20250112 02:57:04.501725 21619 catalog_manager.cc:1559] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:57:04.502120 21619 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:57:04.479786 21359 master_runner.cc:186] Error getting master registration for 127.20.217.189:36049: OK, instance_id { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" instance_seqno: 1736650622426653 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:57:04.511240 21359 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:57:04.513015 21622 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:57:04.519963 21622 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:57:04.543042 21402 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Refusing update from remote peer e76221f7d4ec44da987303d68c393a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:57:04.544745 21618 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:04.573334 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.574045 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:04.583024 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.583865 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:04.584566 21622 catalog_manager.cc:1348] Generated new cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:04.584810 21622 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:57:04.591078 21624 mvcc.cc:204] Tried to move back new op lower bound from 7113320958106148864 to 7113320957866446848. Current Snapshot: MvccSnapshot[applied={T|T < 7113320958106148864}]
I20250112 02:57:04.613369 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:04.614053 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:04.621259 21622 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 02:57:04.625561 21622 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:57:04.654536 21622 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Generated new TSK 0
I20250112 02:57:04.657404 21622 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:57:04.681828 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.129:0
--local_ip_for_outbound_sockets=127.20.217.129
--webserver_interface=127.20.217.129
--webserver_port=0
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:57:04.792536 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } attempt: 7
W20250112 02:57:04.830224 21533 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.217.189:36049 timed out after 0.542s (SENT)
I20250112 02:57:04.861330 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Bootstrap starting.
I20250112 02:57:04.946631 21502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:04.964504 21502 log.cc:826] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:05.009109 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: No bootstrap required, opened a new log
W20250112 02:57:05.209507 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:05.210263 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:05.211292 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:57:05.217876 21636 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:05.220073 21636 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.234737 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:05.237010 21636 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.237157 21502 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.240262 21502 raft_consensus.cc:383] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:05.241315 21502 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ef739ef9dc460ca8287b1d2c6c2217, State: Initialized, Role: FOLLOWER
I20250112 02:57:05.244419 21502 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.263828 21502 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:57:05.264415 21472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:05.265156 21472 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc0757376c3411a9fcbc84ec411befd in term 0.
W20250112 02:57:05.267139 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:05.268654 21628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.129
I20250112 02:57:05.264102 21477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
W20250112 02:57:05.274937 21477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59680 (request call id 1) took 987 ms (client timeout 542 ms). Trace:
I20250112 02:57:05.274955 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
W20250112 02:57:05.275269 21477 rpcz_store.cc:269] 0112 02:57:04.287405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:04.287573 (+ 168us) service_pool.cc:225] Handling call
0112 02:57:05.274865 (+987292us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:05.276158 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:05.277017 21379 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217; no voters: e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:05.277824 21636 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:57:05.278153 21636 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Starting leader election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:05.278452 21636 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:57:05.278492 21475 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:05.272750 21476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:05.272017 21473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:05.284525 21636 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.273593 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
W20250112 02:57:05.286465 21474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59680 (request call id 4) took 447 ms (client timeout 542 ms). Trace:
I20250112 02:57:05.287241 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b"
W20250112 02:57:05.286751 21474 rpcz_store.cc:269] 0112 02:57:04.839274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:04.839396 (+ 122us) service_pool.cc:225] Handling call
0112 02:57:05.286416 (+447020us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 02:57:05.288843 21476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59680 (request call id 2) took 848 ms (client timeout 677 ms). Trace:
W20250112 02:57:05.289151 21476 rpcz_store.cc:269] 0112 02:57:04.440465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:04.440609 (+ 144us) service_pool.cc:225] Handling call
0112 02:57:05.288791 (+848182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:05.289961 21474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:05.291265 21379 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd; no voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:05.293298 21475 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Refusing update from remote peer e76221f7d4ec44da987303d68c393a1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:57:05.293330 21636 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 2 election: Requested vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.293794 21636 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250112 02:57:05.295665 21637 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:05.334692 21628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.217.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.129
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:05.336621 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:05.338692 21628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:57:05.376873 21638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
W20250112 02:57:05.377561 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:05.377642 21638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:05.377000 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:05.378072 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:05.414923 21429 master_runner.cc:232] Existing masters have differing terms: 2 vs 1
I20250112 02:57:05.415402 21429 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:57:05.424505 21660 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:05.424909 21660 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: loading cluster ID for follower catalog manager: success
I20250112 02:57:05.452358 21660 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: acquiring CA information for follower catalog manager: success
I20250112 02:57:05.463377 21660 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:57:05.507539 21619 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:05.507915 21619 catalog_manager.cc:1553] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: loading cluster ID for follower catalog manager: success
I20250112 02:57:05.514979 21619 catalog_manager.cc:1575] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: acquiring CA information for follower catalog manager: success
I20250112 02:57:05.520850 21619 catalog_manager.cc:1603] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:57:05.749286 21401 raft_consensus.cc:1235] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Rejecting Update request from peer e76221f7d4ec44da987303d68c393a1b for earlier term 1. Current term is 2. Ops: []
I20250112 02:57:05.751520 21637 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:57:05.774390 21663 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 LEADER]: Stepping down as leader of term 1
I20250112 02:57:05.775032 21663 raft_consensus.cc:738] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Running, Role: LEADER
I20250112 02:57:05.777921 21663 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.783080 21663 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:57:05.805305 21648 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:05.805799 21648 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.808198 21648 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.809275 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:05.809595 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:05.809916 21556 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9ef739ef9dc460ca8287b1d2c6c2217 in term 2.
I20250112 02:57:05.810355 21401 raft_consensus.cc:2388] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9ef739ef9dc460ca8287b1d2c6c2217 in current term 2: Already voted for candidate bcc0757376c3411a9fcbc84ec411befd in this term.
I20250112 02:57:05.810881 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217; no voters:
I20250112 02:57:05.812134 21648 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:57:05.812449 21648 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:05.812783 21648 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:57:05.819044 21648 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.821483 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd"
I20250112 02:57:05.821707 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b"
I20250112 02:57:05.822197 21401 raft_consensus.cc:2388] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e9ef739ef9dc460ca8287b1d2c6c2217 in current term 2: Already voted for candidate bcc0757376c3411a9fcbc84ec411befd in this term.
I20250112 02:57:05.828145 21556 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9ef739ef9dc460ca8287b1d2c6c2217 in term 2.
I20250112 02:57:05.829064 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd
I20250112 02:57:05.829761 21638 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 02:57:05.842598 21648 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 2 election: Requested vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.850023 21638 raft_consensus.cc:695] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 LEADER]: Becoming Leader. State: Replica: e9ef739ef9dc460ca8287b1d2c6c2217, State: Running, Role: LEADER
I20250112 02:57:05.851318 21638 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.854764 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9ef739ef9dc460ca8287b1d2c6c2217. Latest consensus state: current_term: 2 leader_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:05.855432 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:05.859041 21665 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:05.859484 21665 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.861315 21665 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.862756 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:05.862736 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:05.863494 21556 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc0757376c3411a9fcbc84ec411befd in term 2.
I20250112 02:57:05.865361 21378 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters: e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:05.866257 21665 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 02:57:05.866644 21665 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:05.867000 21665 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:57:05.867323 21666 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:57:05.874022 21666 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:57:05.878579 21666 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:05.879232 21666 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:57:05.879086 21665 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.881510 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:05.885159 21665 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 3 election: Requested vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:05.885962 21666 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:57:05.886112 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b"
I20250112 02:57:05.886660 21556 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:57:05.890116 21666 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Loaded TSK: 0
I20250112 02:57:05.891316 21666 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:57:05.893364 21556 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcc0757376c3411a9fcbc84ec411befd in term 3.
I20250112 02:57:05.894373 21378 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters: e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:05.895068 21665 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 FOLLOWER]: Leader election won for term 3
I20250112 02:57:05.910179 21665 raft_consensus.cc:695] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 LEADER]: Becoming Leader. State: Replica: bcc0757376c3411a9fcbc84ec411befd, State: Running, Role: LEADER
I20250112 02:57:05.911154 21665 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:05.922732 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc0757376c3411a9fcbc84ec411befd. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:05.923494 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:05.933631 21669 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:57:05.944703 21669 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:57:05.948834 21669 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:05.949096 21669 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:57:05.962791 21669 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:57:05.973166 21669 catalog_manager.cc:5910] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Loaded TSK: 0
I20250112 02:57:05.976558 21669 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:57:06.312858 21603 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:57:06.426656 21556 raft_consensus.cc:1235] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Rejecting Update request from peer e9ef739ef9dc460ca8287b1d2c6c2217 for earlier term 2. Current term is 3. Ops: []
I20250112 02:57:06.428848 21670 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250112 02:57:06.436347 21672 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 02:57:06.436739 21672 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9ef739ef9dc460ca8287b1d2c6c2217, State: Running, Role: LEADER
I20250112 02:57:06.437688 21672 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:06.440397 21672 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:57:06.453346 21671 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:06.453933 21671 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:06.457597 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:06.461671 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:06.462692 21671 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:06.478116 21475 raft_consensus.cc:2405] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e76221f7d4ec44da987303d68c393a1b for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20250112 02:57:06.483008 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:06.484629 21671 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:06.527007 21556 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Refusing update from remote peer bcc0757376c3411a9fcbc84ec411befd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250112 02:57:06.529505 21475 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Refusing update from remote peer bcc0757376c3411a9fcbc84ec411befd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250112 02:57:06.529824 21475 pending_rounds.cc:77] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Aborting all ops after (but not including) 4
I20250112 02:57:06.530058 21475 pending_rounds.cc:99] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250112 02:57:06.530285 21475 raft_consensus.cc:2883] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 02:57:06.536229 21673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250112 02:57:06.543561 21674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250112 02:57:06.546980 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc0757376c3411a9fcbc84ec411befd. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.547690 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:06.547912 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc0757376c3411a9fcbc84ec411befd. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.548893 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:06.568419 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.569406 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:06.573546 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.573630 21674 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.574390 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:06.574467 21674 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:06.588390 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:06.589038 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 02:57:05.380016 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:06.761863 21650 debug-util.cc:398] Leaking SignalData structure 0x7b080001c400 after lost signal to thread 21628
W20250112 02:57:07.012557 21628 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 1.639s user 0.525s sys 0.724s
W20250112 02:57:07.015678 21628 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 1.642s user 0.525s sys 0.724s
I20250112 02:57:07.033367 21628 server_base.cc:1034] running on GCE node
W20250112 02:57:07.035296 21683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:07.036810 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:07.050982 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:07.054595 21628 hybrid_clock.cc:648] HybridClock initialized: now 1736650627054519 us; error 83 us; skew 500 ppm
I20250112 02:57:07.055665 21628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:07.075096 21672 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:07.075557 21672 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:07.077380 21672 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:07.078409 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:07.079643 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:07.081205 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:07.081894 21672 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:07.088513 21628 webserver.cc:458] Webserver started at http://127.20.217.129:42869/ using document root <none> and password file <none>
I20250112 02:57:07.089704 21628 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:07.090004 21628 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:07.090560 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:07.097658 21628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data/instance:
uuid: "1c2770701ca54288956df9d910d7d3d2"
format_stamp: "Formatted at 2025-01-12 02:57:07 on dist-test-slave-gppw"
I20250112 02:57:07.099160 21628 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/instance:
uuid: "1c2770701ca54288956df9d910d7d3d2"
format_stamp: "Formatted at 2025-01-12 02:57:07 on dist-test-slave-gppw"
I20250112 02:57:07.125401 21628 fs_manager.cc:696] Time spent creating directory manager: real 0.026s user 0.009s sys 0.016s
I20250112 02:57:07.147527 21688 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:07.148883 21628 fs_manager.cc:730] Time spent opening block manager: real 0.019s user 0.003s sys 0.017s
I20250112 02:57:07.149358 21628 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal
uuid: "1c2770701ca54288956df9d910d7d3d2"
format_stamp: "Formatted at 2025-01-12 02:57:07 on dist-test-slave-gppw"
I20250112 02:57:07.149927 21628 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:07.239374 21628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:07.241348 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:07.241971 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:07.251680 21628 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:57:07.259550 21628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:57:07.259908 21628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:07.260277 21628 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:57:07.260526 21628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:07.535202 21729 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:07.535784 21729 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:07.538576 21729 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:07.538674 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:07.538865 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:07.540472 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:07.541466 21729 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:08.072842 21801 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.129:42085 every 8 connection(s)
I20250112 02:57:08.073356 21628 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.129:42085
I20250112 02:57:08.076857 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
I20250112 02:57:08.082216 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21628
I20250112 02:57:08.082674 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/instance
I20250112 02:57:08.101220 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.130:0
--local_ip_for_outbound_sockets=127.20.217.130
--webserver_interface=127.20.217.130
--webserver_port=0
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:57:08.137517 21802 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:08.137992 21802 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:08.139219 21802 heartbeater.cc:510] Master 127.20.217.189:36049 requested a full tablet report, sending...
I20250112 02:57:08.142633 21467 ts_manager.cc:194] Registered new tserver with Master: 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:08.156533 21805 heartbeater.cc:346] Connected to a master server at 127.20.217.188:35683
I20250112 02:57:08.156891 21805 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:08.157704 21805 heartbeater.cc:510] Master 127.20.217.188:35683 requested a full tablet report, sending...
I20250112 02:57:08.161161 21546 ts_manager.cc:194] Registered new tserver with Master: 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:08.179061 21809 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:08.179437 21809 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:08.180294 21809 heartbeater.cc:510] Master 127.20.217.190:34893 requested a full tablet report, sending...
I20250112 02:57:08.182758 21392 ts_manager.cc:194] Registered new tserver with Master: 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:08.185328 21392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.129:43111
I20250112 02:57:08.508858 21813 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:08.510989 21813 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:08.513625 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:08.516595 21813 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:08.516450 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:08.517879 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:08.518852 21813 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:57:08.591138 21808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:08.591797 21808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:08.592730 21808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:08.651997 21808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:08.653542 21808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.130
I20250112 02:57:08.688638 21809 heartbeater.cc:502] Master 127.20.217.190:34893 was elected leader, sending a full tablet report...
I20250112 02:57:08.715966 21808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.217.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.130
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:08.717552 21808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:08.719494 21808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:08.739246 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:09.532495 21825 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:09.533263 21825 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:09.537473 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:09.537240 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:09.535940 21825 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:09.540288 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:09.541399 21825 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:57:09.790573 21798 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0060 after lost signal to thread 21632
W20250112 02:57:09.792124 21798 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0f40 after lost signal to thread 21801
I20250112 02:57:09.934401 21826 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:09.935354 21826 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:09.939246 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:09.943037 21826 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:09.943234 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:09.944669 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:09.945726 21826 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:57:08.741382 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:10.058624 21808 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 1.319s user 0.468s sys 0.812s
W20250112 02:57:10.059092 21808 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 1.319s user 0.472s sys 0.812s
I20250112 02:57:10.066452 21808 server_base.cc:1034] running on GCE node
W20250112 02:57:10.067571 21828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:10.068917 21808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:10.078958 21808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:10.082542 21808 hybrid_clock.cc:648] HybridClock initialized: now 1736650630082482 us; error 44 us; skew 500 ppm
I20250112 02:57:10.083369 21808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:10.094391 21808 webserver.cc:458] Webserver started at http://127.20.217.130:37843/ using document root <none> and password file <none>
I20250112 02:57:10.095371 21808 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:10.095597 21808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:10.096045 21808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:10.105841 21808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data/instance:
uuid: "cca160b7cffb4ae287ea077e7352ffc9"
format_stamp: "Formatted at 2025-01-12 02:57:10 on dist-test-slave-gppw"
I20250112 02:57:10.107194 21832 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:10.107445 21808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/instance:
uuid: "cca160b7cffb4ae287ea077e7352ffc9"
format_stamp: "Formatted at 2025-01-12 02:57:10 on dist-test-slave-gppw"
I20250112 02:57:10.107693 21832 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:10.110129 21832 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:10.110451 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:10.111270 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:10.113834 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:10.114712 21832 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:10.117760 21808 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.000s sys 0.008s
I20250112 02:57:10.123867 21834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:10.124982 21808 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250112 02:57:10.125294 21808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal
uuid: "cca160b7cffb4ae287ea077e7352ffc9"
format_stamp: "Formatted at 2025-01-12 02:57:10 on dist-test-slave-gppw"
I20250112 02:57:10.125622 21808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:10.253417 21808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:10.255582 21808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:10.256192 21808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:10.259650 21808 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:57:10.265784 21808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:57:10.266076 21808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:10.266397 21808 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:57:10.266664 21808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250112 02:57:10.434759 21808 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.130:36673
I20250112 02:57:10.434895 21946 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.130:36673 every 8 connection(s)
I20250112 02:57:10.437342 21808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
I20250112 02:57:10.445519 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21808
I20250112 02:57:10.446317 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/instance
I20250112 02:57:10.477389 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.131:0
--local_ip_for_outbound_sockets=127.20.217.131
--webserver_interface=127.20.217.131
--webserver_port=0
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:57:10.498046 21949 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:10.498617 21949 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:10.499969 21949 heartbeater.cc:510] Master 127.20.217.190:34893 requested a full tablet report, sending...
I20250112 02:57:10.502565 21392 ts_manager.cc:194] Registered new tserver with Master: cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:10.504221 21392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.130:58729
I20250112 02:57:10.507504 21948 heartbeater.cc:346] Connected to a master server at 127.20.217.188:35683
I20250112 02:57:10.507817 21948 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:10.508636 21948 heartbeater.cc:510] Master 127.20.217.188:35683 requested a full tablet report, sending...
I20250112 02:57:10.511169 21546 ts_manager.cc:194] Registered new tserver with Master: cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:10.514056 21947 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:10.514415 21947 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:10.515341 21947 heartbeater.cc:510] Master 127.20.217.189:36049 requested a full tablet report, sending...
I20250112 02:57:10.518093 21467 ts_manager.cc:194] Registered new tserver with Master: cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
W20250112 02:57:10.817108 21957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:10.817646 21957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:10.818225 21957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:10.848603 21957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:10.849478 21957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.131
I20250112 02:57:10.884119 21957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.217.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.131
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:10.885432 21957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:10.887035 21957 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:10.905750 21964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:11.018970 21949 heartbeater.cc:502] Master 127.20.217.190:34893 was elected leader, sending a full tablet report...
I20250112 02:57:11.027776 21969 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:11.028417 21969 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:11.031610 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:11.032078 21969 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:11.032908 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:11.034297 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:11.035104 21969 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:11.962375 21971 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:11.962924 21971 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:11.964655 21971 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:11.965490 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:11.965879 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:11.967214 21453 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:11.967770 21971 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:57:10.906754 21963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:10.907851 21957 server_base.cc:1034] running on GCE node
W20250112 02:57:10.908423 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:12.096302 21957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:12.098819 21957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:12.100345 21957 hybrid_clock.cc:648] HybridClock initialized: now 1736650632100313 us; error 65 us; skew 500 ppm
I20250112 02:57:12.101089 21957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:12.112915 21957 webserver.cc:458] Webserver started at http://127.20.217.131:37409/ using document root <none> and password file <none>
I20250112 02:57:12.113778 21957 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:12.113961 21957 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:12.114351 21957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:12.119197 21957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data/instance:
uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
format_stamp: "Formatted at 2025-01-12 02:57:12 on dist-test-slave-gppw"
I20250112 02:57:12.120223 21957 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/instance:
uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
format_stamp: "Formatted at 2025-01-12 02:57:12 on dist-test-slave-gppw"
I20250112 02:57:12.127097 21957 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250112 02:57:12.132398 21976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:12.133332 21957 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250112 02:57:12.133626 21957 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal
uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
format_stamp: "Formatted at 2025-01-12 02:57:12 on dist-test-slave-gppw"
I20250112 02:57:12.133941 21957 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:12.165326 21977 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:12.165926 21977 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:12.167939 21977 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:12.168725 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:12.169020 21475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:12.170554 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:12.171635 21977 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:12.199853 21957 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:12.201251 21957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:12.201624 21957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:12.203948 21957 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:57:12.207844 21957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:57:12.208016 21957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:12.208273 21957 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:57:12.208426 21957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:12.336138 21957 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.131:39307
I20250112 02:57:12.336230 22089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.131:39307 every 8 connection(s)
I20250112 02:57:12.338572 21957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
I20250112 02:57:12.340723 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 21957
I20250112 02:57:12.341225 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/instance
I20250112 02:57:12.392596 22090 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:12.393003 22090 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:12.394162 22090 heartbeater.cc:510] Master 127.20.217.189:36049 requested a full tablet report, sending...
I20250112 02:57:12.394892 22091 heartbeater.cc:346] Connected to a master server at 127.20.217.188:35683
I20250112 02:57:12.395056 22092 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:12.395411 22092 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:12.395458 22091 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:12.396131 22092 heartbeater.cc:510] Master 127.20.217.190:34893 requested a full tablet report, sending...
I20250112 02:57:12.396363 22091 heartbeater.cc:510] Master 127.20.217.188:35683 requested a full tablet report, sending...
I20250112 02:57:12.398059 21392 ts_manager.cc:194] Registered new tserver with Master: e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:12.398066 21467 ts_manager.cc:194] Registered new tserver with Master: e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:12.399343 21546 ts_manager.cc:194] Registered new tserver with Master: e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:12.399819 21392 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.131:60225
I20250112 02:57:12.407805 21350 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 02:57:12.467432 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_1b01220e389042338f97183d2738b0bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:12.471227 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b6b3ecdf31974453bf5e93f2b6c0752f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:12.472754 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6b3ecdf31974453bf5e93f2b6c0752f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:12.475834 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_0255a897c32046f9b1e52bfbe2517388"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:12.480170 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a66f18b2f9e2494e94b0cb8f0d583273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:12.485668 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0255a897c32046f9b1e52bfbe2517388 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:12.482311 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a66f18b2f9e2494e94b0cb8f0d583273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:12.486693 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b01220e389042338f97183d2738b0bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:12.540107 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b6b3ecdf31974453bf5e93f2b6c0752f" } new_table_name: "default.Table_7d28d4008d564d7fa366aacbc6922b33" modify_external_catalogs: true
I20250112 02:57:12.555397 21390 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a66f18b2f9e2494e94b0cb8f0d583273" } modify_external_catalogs: true
I20250112 02:57:12.556051 21390 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a66f18b2f9e2494e94b0cb8f0d583273" } modify_external_catalogs: true
I20250112 02:57:12.559319 21389 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_0255a897c32046f9b1e52bfbe2517388" } new_table_name: "default.Table_9685195d050c46269d0109dcc9157a2b" modify_external_catalogs: true
I20250112 02:57:12.563230 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_1b01220e389042338f97183d2738b0bd" } modify_external_catalogs: true
I20250112 02:57:12.563742 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_1b01220e389042338f97183d2738b0bd" } modify_external_catalogs: true
W20250112 02:57:12.597560 21388 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38a1cb220c074436850aa3dc6df518cc consensus state has no leader
I20250112 02:57:12.636445 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 38a1cb220c074436850aa3dc6df518cc (DEFAULT_TABLE table=default.Table_b6b3ecdf31974453bf5e93f2b6c0752f [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.638845 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a1cb220c074436850aa3dc6df518cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.640215 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 38a1cb220c074436850aa3dc6df518cc (DEFAULT_TABLE table=default.Table_b6b3ecdf31974453bf5e93f2b6c0752f [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.642061 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a1cb220c074436850aa3dc6df518cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:12.643921 21389 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 750dbf8e68cb457e8871641582c9ba27 has no consensus state
I20250112 02:57:12.647678 21392 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d6620dc1ebfb4c099097486702a11881
I20250112 02:57:12.649073 21390 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8c61b6e4239e4d80973fdca98bd7d69f
W20250112 02:57:12.659559 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 38a1cb220c074436850aa3dc6df518cc (table default.Table_7d28d4008d564d7fa366aacbc6922b33 [id=e2c68fb77f964468905c2b7b699408b5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 38a1cb220c074436850aa3dc6df518cc consensus state has no leader
I20250112 02:57:12.663347 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 38a1cb220c074436850aa3dc6df518cc (DEFAULT_TABLE table=default.Table_b6b3ecdf31974453bf5e93f2b6c0752f [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.666280 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21429
I20250112 02:57:12.673660 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38a1cb220c074436850aa3dc6df518cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.680753 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 750dbf8e68cb457e8871641582c9ba27 (DEFAULT_TABLE table=default.Table_9685195d050c46269d0109dcc9157a2b [id=67a9c7c8457e4345ac77ab0e82bd682a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.681253 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 750dbf8e68cb457e8871641582c9ba27 (DEFAULT_TABLE table=default.Table_9685195d050c46269d0109dcc9157a2b [id=67a9c7c8457e4345ac77ab0e82bd682a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.681946 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750dbf8e68cb457e8871641582c9ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.682248 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750dbf8e68cb457e8871641582c9ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.684759 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 750dbf8e68cb457e8871641582c9ba27 (DEFAULT_TABLE table=default.Table_9685195d050c46269d0109dcc9157a2b [id=67a9c7c8457e4345ac77ab0e82bd682a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.685945 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750dbf8e68cb457e8871641582c9ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.713496 22143 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:12.728885 21390 master_service.cc:946] ReplaceTablet: received request to replace tablet 38a1cb220c074436850aa3dc6df518cc from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:12.730126 22145 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:12.736212 21379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.217.189:36049 (error 108) [suppressed 12 similar messages]
W20250112 02:57:12.738363 21692 connection.cc:537] client connection to 127.20.217.189:36049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:12.738909 22144 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:12.740114 22143 tablet_bootstrap.cc:654] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:12.740669 21802 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:36049 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:12.742529 22143 log.cc:826] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:12.746241 22145 tablet_bootstrap.cc:654] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:12.746814 22144 tablet_bootstrap.cc:654] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:12.749176 22144 log.cc:826] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:12.749187 22145 log.cc:826] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:12.759009 22144 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:12.759567 22144 ts_tablet_manager.cc:1397] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.006s
I20250112 02:57:12.762065 21390 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 38a1cb220c074436850aa3dc6df518cc
I20250112 02:57:12.763944 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:12.766065 22143 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:12.766631 22143 ts_tablet_manager.cc:1397] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.054s user 0.015s sys 0.005s
I20250112 02:57:12.771306 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:12.775075 21390 catalog_manager.cc:6596] ReplaceTablet: tablet 38a1cb220c074436850aa3dc6df518cc deleted and replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:12.779392 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:12.784911 22145 tablet_bootstrap.cc:492] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:12.785434 22145 ts_tablet_manager.cc:1397] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.007s
W20250112 02:57:12.775712 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 38a1cb220c074436850aa3dc6df518cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a1cb220c074436850aa3dc6df518cc already in progress: creating tablet
I20250112 02:57:12.793311 22144 raft_consensus.cc:357] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:12.794270 22144 raft_consensus.cc:383] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:12.794581 22144 raft_consensus.cc:738] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:12.795572 22144 consensus_queue.cc:260] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
W20250112 02:57:12.799188 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 38a1cb220c074436850aa3dc6df518cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a1cb220c074436850aa3dc6df518cc already in progress: creating tablet
W20250112 02:57:12.801359 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 38a1cb220c074436850aa3dc6df518cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a1cb220c074436850aa3dc6df518cc already in progress: creating tablet
I20250112 02:57:12.805438 22144 ts_tablet_manager.cc:1428] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.046s user 0.029s sys 0.011s
W20250112 02:57:12.813040 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 76977.9us
I20250112 02:57:12.816828 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_6b73091900e44fbf8fd230b259f7b8a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:12.818833 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b73091900e44fbf8fd230b259f7b8a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:12.820060 22144 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:12.822733 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:12.823562 22153 tablet_replica.cc:331] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:12.824376 22153 raft_consensus.cc:2238] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:12.824960 22153 raft_consensus.cc:2267] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:12.829609 21389 master_service.cc:946] ReplaceTablet: received request to replace tablet 38a1cb220c074436850aa3dc6df518cc from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:12.831610 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_33f043ad58cf4595a34feee4e8aad678"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:12.833398 22153 ts_tablet_manager.cc:1905] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:12.833353 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f043ad58cf4595a34feee4e8aad678 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:12.848278 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_4c0ff8b26e54463c9f0c92213bd77228"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:12.850965 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c0ff8b26e54463c9f0c92213bd77228 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:12.855319 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:12.855243 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_81bcb9239f4b4a6faa019636bd9b7c2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:12.857030 21387 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_9685195d050c46269d0109dcc9157a2b" } new_table_name: "default.Table_27a4219bed9146b78a55c9ff1ab520b9" modify_external_catalogs: true
W20250112 02:57:12.857344 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81bcb9239f4b4a6faa019636bd9b7c2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:12.855232 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7d28d4008d564d7fa366aacbc6922b33" } new_table_name: "default.Table_97850d925a00442ab1e1296aa4c87e45" modify_external_catalogs: true
I20250112 02:57:12.862581 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (DEFAULT_TABLE table=default.Table_7d28d4008d564d7fa366aacbc6922b33 [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.863598 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.864663 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (DEFAULT_TABLE table=default.Table_7d28d4008d564d7fa366aacbc6922b33 [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:12.865342 21947 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:36049 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:12.865706 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.867697 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (DEFAULT_TABLE table=default.Table_7d28d4008d564d7fa366aacbc6922b33 [id=e2c68fb77f964468905c2b7b699408b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:12.868736 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:12.869302 22144 tablet_bootstrap.cc:654] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:12.883505 22153 ts_tablet_manager.cc:1918] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:12.900040 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_6b73091900e44fbf8fd230b259f7b8a2" } modify_external_catalogs: true
I20250112 02:57:12.900527 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_6b73091900e44fbf8fd230b259f7b8a2" } modify_external_catalogs: true
I20250112 02:57:12.902434 22092 heartbeater.cc:502] Master 127.20.217.190:34893 was elected leader, sending a full tablet report...
I20250112 02:57:12.899962 21389 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_33f043ad58cf4595a34feee4e8aad678" } modify_external_catalogs: true
I20250112 02:57:12.906386 21389 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_33f043ad58cf4595a34feee4e8aad678" } modify_external_catalogs: true
I20250112 02:57:12.909639 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_9685195d050c46269d0109dcc9157a2b [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 1 got 0
I20250112 02:57:12.910290 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (table default.Table_7d28d4008d564d7fa366aacbc6922b33 [id=e2c68fb77f964468905c2b7b699408b5]). Expected version 1 got 0
W20250112 02:57:12.914562 22090 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:36049 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
W20250112 02:57:12.935928 21387 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 750dbf8e68cb457e8871641582c9ba27 consensus state has no leader
I20250112 02:57:12.939942 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:12.940851 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 consensus state has no leader
W20250112 02:57:12.951658 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 38a1cb220c074436850aa3dc6df518cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38a1cb220c074436850aa3dc6df518cc already in progress: creating tablet
I20250112 02:57:12.954676 22153 log.cc:1198] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/38a1cb220c074436850aa3dc6df518cc
I20250112 02:57:12.955622 22153 ts_tablet_manager.cc:1939] T 38a1cb220c074436850aa3dc6df518cc P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:12.958508 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 38a1cb220c074436850aa3dc6df518cc (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
I20250112 02:57:12.979404 21389 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8b5b92d006304a0baa51370d59644f29
I20250112 02:57:12.983932 22144 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:12.984364 22144 ts_tablet_manager.cc:1397] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.165s user 0.006s sys 0.012s
I20250112 02:57:12.986994 22144 raft_consensus.cc:357] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:12.987753 22144 raft_consensus.cc:383] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:12.988070 22144 raft_consensus.cc:738] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:12.989084 22144 consensus_queue.cc:260] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:12.998594 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_27a4219bed9146b78a55c9ff1ab520b9 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 2 got 0
I20250112 02:57:13.000473 22144 ts_tablet_manager.cc:1428] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.016s user 0.002s sys 0.005s
I20250112 02:57:13.015007 22143 raft_consensus.cc:357] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.015954 22144 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:13.015980 22143 raft_consensus.cc:383] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.016264 22143 raft_consensus.cc:738] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.020645 21392 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 06e3ec081ed743ac84e94806da64afc5
I20250112 02:57:13.024693 22144 tablet_bootstrap.cc:654] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.035254 22144 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:13.035643 22144 ts_tablet_manager.cc:1397] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20250112 02:57:13.038069 22144 raft_consensus.cc:357] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.038816 22144 raft_consensus.cc:383] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.017150 22143 consensus_queue.cc:260] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.039095 22144 raft_consensus.cc:738] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.039722 22144 consensus_queue.cc:260] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.043625 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_27a4219bed9146b78a55c9ff1ab520b9 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 2 got 0
I20250112 02:57:13.045962 22144 ts_tablet_manager.cc:1428] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250112 02:57:13.048099 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b (DEFAULT_TABLE table=default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.049124 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdb1b8b1db746abb082a1039ee9268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.051185 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b (DEFAULT_TABLE table=default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.052147 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdb1b8b1db746abb082a1039ee9268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.056208 22143 ts_tablet_manager.cc:1428] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.289s user 0.032s sys 0.006s
I20250112 02:57:13.058193 22143 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.065420 22143 tablet_bootstrap.cc:654] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.069859 21392 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]). Expected version 2 got 0
I20250112 02:57:13.072232 22145 raft_consensus.cc:357] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.073207 22145 raft_consensus.cc:383] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.073513 22145 raft_consensus.cc:738] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.074414 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b (DEFAULT_TABLE table=default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.075917 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fdb1b8b1db746abb082a1039ee9268b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.075791 22145 consensus_queue.cc:260] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.090840 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:13.091817 22151 tablet_replica.cc:331] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:13.092588 22151 raft_consensus.cc:2238] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.093152 22151 raft_consensus.cc:2267] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.096663 22145 ts_tablet_manager.cc:1428] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.311s user 0.038s sys 0.005s
I20250112 02:57:13.098774 22151 ts_tablet_manager.cc:1905] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.111389 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_86735ff40c83478eafeb3d073a86f588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.113081 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a1cb220c074436850aa3dc6df518cc with delete_type TABLET_DATA_DELETED (replaced by tablet 3497ff26cfbe4a73b1f7bb54b83a32a7) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:13.113328 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_60e29c461b3f4842aecb00c8d23d169f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.113969 22152 tablet_replica.cc:331] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:13.114867 22152 raft_consensus.cc:2238] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.115438 22152 raft_consensus.cc:2267] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.115291 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_f373485980284f01b5f3f93ff6866d33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.122992 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (DEFAULT_TABLE table=default.Table_81bcb9239f4b4a6faa019636bd9b7c2f [id=1308d8a9ae7d485489daf139f579b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.124675 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fbebff35c344bdb305ea1a6dbbefe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.125941 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_0162c6f72cb74aa9a0e65d704255d604"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.129747 22151 ts_tablet_manager.cc:1918] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:57:13.130014 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f373485980284f01b5f3f93ff6866d33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.130187 22151 log.cc:1198] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/38a1cb220c074436850aa3dc6df518cc
I20250112 02:57:13.131067 22151 ts_tablet_manager.cc:1939] T 38a1cb220c074436850aa3dc6df518cc P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
W20250112 02:57:13.134048 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86735ff40c83478eafeb3d073a86f588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.133750 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (DEFAULT_TABLE table=default.Table_81bcb9239f4b4a6faa019636bd9b7c2f [id=1308d8a9ae7d485489daf139f579b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.135177 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fbebff35c344bdb305ea1a6dbbefe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.135555 22145 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:13.135761 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0162c6f72cb74aa9a0e65d704255d604 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:13.134819 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60e29c461b3f4842aecb00c8d23d169f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.137403 22152 ts_tablet_manager.cc:1905] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.139706 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 38a1cb220c074436850aa3dc6df518cc (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
I20250112 02:57:13.141041 22145 tablet_bootstrap.cc:654] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.147687 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (DEFAULT_TABLE table=default.Table_81bcb9239f4b4a6faa019636bd9b7c2f [id=1308d8a9ae7d485489daf139f579b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.148777 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fbebff35c344bdb305ea1a6dbbefe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.146417 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_81bcb9239f4b4a6faa019636bd9b7c2f" } new_table_name: "default.Table_91e4fb97a4254b07b6f2f09c206c51a5" modify_external_catalogs: true
I20250112 02:57:13.150799 22143 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:13.151368 22143 ts_tablet_manager.cc:1397] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.008s
I20250112 02:57:13.154953 22143 raft_consensus.cc:357] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.155709 22152 ts_tablet_manager.cc:1918] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.155980 22143 raft_consensus.cc:383] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.156143 22152 log.cc:1198] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/38a1cb220c074436850aa3dc6df518cc
I20250112 02:57:13.156359 22143 raft_consensus.cc:738] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.167268 22152 ts_tablet_manager.cc:1939] T 38a1cb220c074436850aa3dc6df518cc P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:13.170550 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_27a4219bed9146b78a55c9ff1ab520b9" } new_table_name: "default.Table_f24d095676be44b9bee7d9655f49fb22" modify_external_catalogs: true
I20250112 02:57:13.171391 22143 consensus_queue.cc:260] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.176198 22144 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:13.176332 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 38a1cb220c074436850aa3dc6df518cc (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
I20250112 02:57:13.177088 22143 ts_tablet_manager.cc:1428] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.025s user 0.008s sys 0.001s
I20250112 02:57:13.178901 22143 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.182089 22144 tablet_bootstrap.cc:654] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.182478 21389 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_27a4219bed9146b78a55c9ff1ab520b9 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 2 got 0
I20250112 02:57:13.186362 22143 tablet_bootstrap.cc:654] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.189303 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_4c0ff8b26e54463c9f0c92213bd77228" } modify_external_catalogs: true
I20250112 02:57:13.194346 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:13.196344 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:13.190069 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_4c0ff8b26e54463c9f0c92213bd77228" } modify_external_catalogs: true
W20250112 02:57:13.217373 21954 tablet.cc:2367] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:57:13.231137 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_97850d925a00442ab1e1296aa4c87e45" } modify_external_catalogs: true
I20250112 02:57:13.231817 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_97850d925a00442ab1e1296aa4c87e45" } modify_external_catalogs: true
W20250112 02:57:13.248786 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 750dbf8e68cb457e8871641582c9ba27 consensus state has no leader
I20250112 02:57:13.256736 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_f24d095676be44b9bee7d9655f49fb22 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 3 got 0
I20250112 02:57:13.290427 22145 tablet_bootstrap.cc:492] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:13.291224 22145 ts_tablet_manager.cc:1397] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.156s user 0.026s sys 0.026s
I20250112 02:57:13.295308 22145 raft_consensus.cc:357] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:13.297652 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11fbebff35c344bdb305ea1a6dbbefe4 consensus state has no leader
I20250112 02:57:13.298813 22145 raft_consensus.cc:383] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.299250 22145 raft_consensus.cc:738] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.302116 22145 consensus_queue.cc:260] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.306577 22145 ts_tablet_manager.cc:1428] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20250112 02:57:13.307847 22145 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:13.316990 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_f24d095676be44b9bee7d9655f49fb22 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 3 got 0
I20250112 02:57:13.326388 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_f24d095676be44b9bee7d9655f49fb22 [id=67a9c7c8457e4345ac77ab0e82bd682a]). Expected version 3 got 0
I20250112 02:57:13.329149 22145 tablet_bootstrap.cc:654] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.336340 22144 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:13.336767 22144 ts_tablet_manager.cc:1397] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.161s user 0.046s sys 0.065s
I20250112 02:57:13.339282 22144 raft_consensus.cc:357] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.340021 22144 raft_consensus.cc:383] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.340278 22144 raft_consensus.cc:738] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.340993 22144 consensus_queue.cc:260] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.347510 22144 ts_tablet_manager.cc:1428] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250112 02:57:13.350802 22143 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:13.351243 22143 ts_tablet_manager.cc:1397] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.173s user 0.013s sys 0.059s
I20250112 02:57:13.353850 22143 raft_consensus.cc:357] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.354600 22143 raft_consensus.cc:383] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.354880 22143 raft_consensus.cc:738] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.355516 22143 consensus_queue.cc:260] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.359943 22143 ts_tablet_manager.cc:1428] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 02:57:13.360414 22144 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:13.360774 22143 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.366396 22144 tablet_bootstrap.cc:654] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:13.367897 22097 tablet.cc:2367] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:57:13.372820 22145 tablet_bootstrap.cc:492] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:13.373266 22145 ts_tablet_manager.cc:1397] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.011s
I20250112 02:57:13.376305 22145 raft_consensus.cc:357] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.377029 22145 raft_consensus.cc:383] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.377303 22145 raft_consensus.cc:738] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.377975 22145 consensus_queue.cc:260] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.379932 22143 tablet_bootstrap.cc:654] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.382231 22154 raft_consensus.cc:491] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:13.382936 22154 raft_consensus.cc:513] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.390868 22154 leader_election.cc:290] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:13.395539 21812 tablet.cc:2367] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:57:13.403424 22163 raft_consensus.cc:491] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:13.404114 22163 raft_consensus.cc:513] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.409533 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 1fdb1b8b1db746abb082a1039ee9268b
I20250112 02:57:13.410977 22163 leader_election.cc:290] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:13.411787 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:13.413067 22145 ts_tablet_manager.cc:1428] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.040s user 0.004s sys 0.004s
I20250112 02:57:13.413919 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:13.415613 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 1fdb1b8b1db746abb082a1039ee9268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fdb1b8b1db746abb082a1039ee9268b already in progress: creating tablet
W20250112 02:57:13.416572 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 1fdb1b8b1db746abb082a1039ee9268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fdb1b8b1db746abb082a1039ee9268b already in progress: creating tablet
I20250112 02:57:13.419798 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:13.421705 22145 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:13.422032 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 1fdb1b8b1db746abb082a1039ee9268b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fdb1b8b1db746abb082a1039ee9268b already in progress: creating tablet
I20250112 02:57:13.447886 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_dec32526aa174f35b386fb5ccc07c740"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:13.450065 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec32526aa174f35b386fb5ccc07c740 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.452723 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f24d095676be44b9bee7d9655f49fb22" } new_table_name: "default.Table_3cbd6af691c1497bb9829994c20e0645" modify_external_catalogs: true
I20250112 02:57:13.460757 22143 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:13.461413 22143 ts_tablet_manager.cc:1397] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.009s
I20250112 02:57:13.477070 22154 raft_consensus.cc:491] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:13.477535 22154 raft_consensus.cc:513] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.479498 22145 tablet_bootstrap.cc:654] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.479861 22154 leader_election.cc:290] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:13.491307 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 750dbf8e68cb457e8871641582c9ba27 consensus state has no leader
I20250112 02:57:13.491171 22143 raft_consensus.cc:357] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.495921 22143 raft_consensus.cc:383] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.497938 21736 tablet_service.cc:1467] Processing CreateTablet for tablet b1e608afe71349f1ae609e584022e510 (DEFAULT_TABLE table=default.Table_f373485980284f01b5f3f93ff6866d33 [id=32f1cf819ea748d78535c9c44d64b92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.498441 21384 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:13.499547 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e608afe71349f1ae609e584022e510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.499549 22024 tablet_service.cc:1467] Processing CreateTablet for tablet b1e608afe71349f1ae609e584022e510 (DEFAULT_TABLE table=default.Table_f373485980284f01b5f3f93ff6866d33 [id=32f1cf819ea748d78535c9c44d64b92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.500579 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e608afe71349f1ae609e584022e510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.504739 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:13.505435 22153 tablet_replica.cc:331] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:13.506151 22153 raft_consensus.cc:2238] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.506366 22161 raft_consensus.cc:491] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:13.506624 22153 raft_consensus.cc:2267] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.506889 22161 raft_consensus.cc:513] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:13.507241 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497ff26cfbe4a73b1f7bb54b83a32a7" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:13.507627 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750dbf8e68cb457e8871641582c9ba27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:13.508065 22045 raft_consensus.cc:2463] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:13.508222 22044 raft_consensus.cc:2463] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:13.509452 21836 leader_election.cc:304] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:13.510134 21836 raft_consensus.cc:2681] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:57:13.510244 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_81a50ebd3f5c40dd9893310926f5439e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:13.512518 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81a50ebd3f5c40dd9893310926f5439e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.513736 22161 leader_election.cc:290] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:13.513835 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:13.514554 22152 tablet_replica.cc:331] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:13.515266 22152 raft_consensus.cc:2238] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.515739 22152 raft_consensus.cc:2267] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.517038 21836 leader_election.cc:304] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:13.517835 22152 ts_tablet_manager.cc:1905] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.517848 22154 raft_consensus.cc:2798] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:13.517782 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_91e4fb97a4254b07b6f2f09c206c51a5" } new_table_name: "default.Table_38868f576fce4ebcb97f42b8f83724e8" modify_external_catalogs: true
I20250112 02:57:13.518204 22154 raft_consensus.cc:491] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:13.518537 22154 raft_consensus.cc:3054] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:13.523092 22153 ts_tablet_manager.cc:1905] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.536586 22143 raft_consensus.cc:738] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.537446 22143 consensus_queue.cc:260] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.540038 22143 ts_tablet_manager.cc:1428] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.078s user 0.002s sys 0.007s
I20250112 02:57:13.540949 22143 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.552227 22152 ts_tablet_manager.cc:1918] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.552655 22152 log.cc:1198] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:13.553277 22152 ts_tablet_manager.cc:1939] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:13.569217 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_34e82e7a08e34857a67138d298e2862e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.569566 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
W20250112 02:57:13.571751 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e82e7a08e34857a67138d298e2862e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.581758 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:13.582798 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497ff26cfbe4a73b1f7bb54b83a32a7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:13.583544 21757 raft_consensus.cc:2463] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:13.584241 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:13.584322 22151 tablet_replica.cc:331] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:13.585264 21978 leader_election.cc:304] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:13.585484 22151 raft_consensus.cc:2238] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.585927 22151 raft_consensus.cc:2267] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:13.586046 21978 raft_consensus.cc:2681] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:57:13.587932 22151 ts_tablet_manager.cc:1905] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.589161 22145 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:13.593986 22145 ts_tablet_manager.cc:1397] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.172s user 0.010s sys 0.007s
I20250112 02:57:13.603219 21387 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_91e4fb97a4254b07b6f2f09c206c51a5 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 1 got 0
I20250112 02:57:13.605290 22145 raft_consensus.cc:357] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.607654 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e (DEFAULT_TABLE table=default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.608654 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737bb9fd0c324e0d8ba7e4bc373be80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.609516 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba (DEFAULT_TABLE table=default.Table_60e29c461b3f4842aecb00c8d23d169f [id=417cdf8adb1841e1a3793f3db4e112cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.607894 22145 raft_consensus.cc:383] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.610123 22145 raft_consensus.cc:738] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.610755 22145 consensus_queue.cc:260] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:13.612423 21690 leader_election.cc:343] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:13.612567 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e (DEFAULT_TABLE table=default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.611776 22143 tablet_bootstrap.cc:654] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.612550 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3af671bcea480a8f48b3c4abcc74ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.613109 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba (DEFAULT_TABLE table=default.Table_60e29c461b3f4842aecb00c8d23d169f [id=417cdf8adb1841e1a3793f3db4e112cc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:13.613344 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11fbebff35c344bdb305ea1a6dbbefe4 consensus state has no leader
I20250112 02:57:13.615108 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3af671bcea480a8f48b3c4abcc74ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.613241 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b3ff88d1930d4b40bf0740d55d0e949b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.606658 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 (DEFAULT_TABLE table=default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.619275 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faf24cad2f0456caa5a151a1a11a5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.621029 22144 tablet_bootstrap.cc:492] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:13.621417 22144 ts_tablet_manager.cc:1397] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.261s user 0.031s sys 0.016s
I20250112 02:57:13.623971 22144 raft_consensus.cc:357] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.624851 22144 raft_consensus.cc:383] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.618919 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_34e82e7a08e34857a67138d298e2862e" } modify_external_catalogs: true
I20250112 02:57:13.631108 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_34e82e7a08e34857a67138d298e2862e" } modify_external_catalogs: true
I20250112 02:57:13.622359 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 (DEFAULT_TABLE table=default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.624322 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497ff26cfbe4a73b1f7bb54b83a32a7" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:13.636072 21757 raft_consensus.cc:2463] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:13.625545 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750dbf8e68cb457e8871641582c9ba27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:13.637364 21756 raft_consensus.cc:2463] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
W20250112 02:57:13.639014 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3ff88d1930d4b40bf0740d55d0e949b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.642325 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faf24cad2f0456caa5a151a1a11a5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.644004 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737bb9fd0c324e0d8ba7e4bc373be80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.660724 21881 tablet_service.cc:1467] Processing CreateTablet for tablet b1e608afe71349f1ae609e584022e510 (DEFAULT_TABLE table=default.Table_f373485980284f01b5f3f93ff6866d33 [id=32f1cf819ea748d78535c9c44d64b92e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:13.661166 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 11fbebff35c344bdb305ea1a6dbbefe4 consensus state has no leader
I20250112 02:57:13.661811 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1e608afe71349f1ae609e584022e510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.671715 22145 ts_tablet_manager.cc:1428] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.069s user 0.006s sys 0.000s
I20250112 02:57:13.672675 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:13.673352 22152 tablet_replica.cc:331] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:13.674022 22152 raft_consensus.cc:2238] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.674491 22152 raft_consensus.cc:2267] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.625157 22144 raft_consensus.cc:738] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.677099 21384 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e254cd24ac71433aa66da856e9554329
I20250112 02:57:13.680127 22152 ts_tablet_manager.cc:1905] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.684535 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 (DEFAULT_TABLE table=default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.685966 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6faf24cad2f0456caa5a151a1a11a5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.686933 22151 ts_tablet_manager.cc:1918] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.687916 22151 log.cc:1198] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/1fdb1b8b1db746abb082a1039ee9268b
I20250112 02:57:13.688707 22151 ts_tablet_manager.cc:1939] T 1fdb1b8b1db746abb082a1039ee9268b P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:13.695350 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 1fdb1b8b1db746abb082a1039ee9268b (table default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]) successfully deleted
I20250112 02:57:13.696295 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f373485980284f01b5f3f93ff6866d33" } new_table_name: "default.Table_81e59853126a44968bc4c56e97efc6bc" modify_external_catalogs: true
I20250112 02:57:13.697381 22154 raft_consensus.cc:513] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.699595 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e (DEFAULT_TABLE table=default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.701514 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 737bb9fd0c324e0d8ba7e4bc373be80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.674108 22145 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:13.682736 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba (DEFAULT_TABLE table=default.Table_60e29c461b3f4842aecb00c8d23d169f [id=417cdf8adb1841e1a3793f3db4e112cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.703233 22151 tablet_replica.cc:331] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:13.690479 22144 consensus_queue.cc:260] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.703975 22151 raft_consensus.cc:2238] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:13.705202 22151 raft_consensus.cc:2267] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:13.706477 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3af671bcea480a8f48b3c4abcc74ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.707718 21385 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 2 got 0
I20250112 02:57:13.711205 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 06f24063285e4b32989514ab8863fb14 (DEFAULT_TABLE table=default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.712195 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f24063285e4b32989514ab8863fb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.712029 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b3ff88d1930d4b40bf0740d55d0e949b" } modify_external_catalogs: true
I20250112 02:57:13.712525 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b3ff88d1930d4b40bf0740d55d0e949b" } modify_external_catalogs: true
I20250112 02:57:13.711012 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 06f24063285e4b32989514ab8863fb14 (DEFAULT_TABLE table=default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.722191 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750dbf8e68cb457e8871641582c9ba27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:13.722730 21756 raft_consensus.cc:3054] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:13.723155 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f24063285e4b32989514ab8863fb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.727725 21733 tablet_service.cc:1467] Processing CreateTablet for tablet a90643c97b5e424a8bf3c262e3e5429e (DEFAULT_TABLE table=default.Table_81a50ebd3f5c40dd9893310926f5439e [id=14a6d904311440469400b09be94d6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.729084 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90643c97b5e424a8bf3c262e3e5429e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:13.731098 21388 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1e608afe71349f1ae609e584022e510 consensus state has no leader
I20250112 02:57:13.734511 22143 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:13.734942 22143 ts_tablet_manager.cc:1397] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.194s user 0.006s sys 0.007s
I20250112 02:57:13.737387 22143 raft_consensus.cc:357] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.738065 21756 raft_consensus.cc:2463] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:13.740805 22151 ts_tablet_manager.cc:1905] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:13.746975 22154 leader_election.cc:290] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:13.748378 21385 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 27429ddcc02140898745b324758f18c9
I20250112 02:57:13.748392 21835 leader_election.cc:304] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:13.750391 21877 tablet_service.cc:1467] Processing CreateTablet for tablet a90643c97b5e424a8bf3c262e3e5429e (DEFAULT_TABLE table=default.Table_81a50ebd3f5c40dd9893310926f5439e [id=14a6d904311440469400b09be94d6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.747097 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497ff26cfbe4a73b1f7bb54b83a32a7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:13.749737 22154 raft_consensus.cc:2798] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:13.757375 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 06f24063285e4b32989514ab8863fb14 (DEFAULT_TABLE table=default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.762359 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90643c97b5e424a8bf3c262e3e5429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.767242 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_419f8e3cdd474bdfbcead2ec83af6032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:13.768719 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f24063285e4b32989514ab8863fb14. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:13.768923 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_419f8e3cdd474bdfbcead2ec83af6032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.771093 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_c41d4357625c47ccb556d7a2231c65fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:13.772639 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c41d4357625c47ccb556d7a2231c65fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.772707 22151 ts_tablet_manager.cc:1918] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.773121 22151 log.cc:1198] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:13.773725 22151 ts_tablet_manager.cc:1939] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:13.777148 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
I20250112 02:57:13.738091 22143 raft_consensus.cc:383] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.779917 22143 raft_consensus.cc:738] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.780553 22143 consensus_queue.cc:260] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.782889 22143 ts_tablet_manager.cc:1428] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.048s user 0.003s sys 0.004s
I20250112 02:57:13.783699 22143 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.785851 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 2 got 0
I20250112 02:57:13.791112 22143 tablet_bootstrap.cc:654] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.792546 21387 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 2 got 0
I20250112 02:57:13.767410 22154 raft_consensus.cc:695] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:13.814627 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_f6351a54c2544bfe94a9b4c219f69ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:13.816325 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6351a54c2544bfe94a9b4c219f69ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.821971 22144 ts_tablet_manager.cc:1428] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.200s user 0.010s sys 0.005s
I20250112 02:57:13.830811 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_419f8e3cdd474bdfbcead2ec83af6032" } new_table_name: "default.Table_be3f6ab8fc684025b5b231ca16c14c9e" modify_external_catalogs: true
I20250112 02:57:13.841152 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750dbf8e68cb457e8871641582c9ba27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:13.842003 22044 raft_consensus.cc:3054] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:13.844569 21876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fdb1b8b1db746abb082a1039ee9268b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:13.844905 21980 leader_election.cc:343] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:57:13.847602 22022 tablet_service.cc:1467] Processing CreateTablet for tablet a90643c97b5e424a8bf3c262e3e5429e (DEFAULT_TABLE table=default.Table_81a50ebd3f5c40dd9893310926f5439e [id=14a6d904311440469400b09be94d6748]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.849246 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a90643c97b5e424a8bf3c262e3e5429e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.840420 22154 consensus_queue.cc:237] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:13.862715 22143 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:13.863147 22143 ts_tablet_manager.cc:1397] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.080s user 0.025s sys 0.028s
I20250112 02:57:13.865721 22143 raft_consensus.cc:357] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.866411 22143 raft_consensus.cc:383] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.866693 22143 raft_consensus.cc:738] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:13.867403 22143 consensus_queue.cc:260] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.872664 22143 ts_tablet_manager.cc:1428] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20250112 02:57:13.879149 22143 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:13.879881 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 (DEFAULT_TABLE table=default.Table_419f8e3cdd474bdfbcead2ec83af6032 [id=1322b48e2bf14cf2851b7f0e1828441c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.880887 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f77f8b613544e909ed3397c4b2ebd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.881829 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_81e59853126a44968bc4c56e97efc6bc [id=32f1cf819ea748d78535c9c44d64b92e]). Expected version 1 got 0
I20250112 02:57:13.883533 21387 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3cbd6af691c1497bb9829994c20e0645" } modify_external_catalogs: true
I20250112 02:57:13.884073 21387 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3cbd6af691c1497bb9829994c20e0645" } modify_external_catalogs: true
I20250112 02:57:13.889453 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_81e59853126a44968bc4c56e97efc6bc [id=32f1cf819ea748d78535c9c44d64b92e]). Expected version 1 got 0
I20250112 02:57:13.898738 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3497ff26cfbe4a73b1f7bb54b83a32a7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:13.903645 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 (DEFAULT_TABLE table=default.Table_419f8e3cdd474bdfbcead2ec83af6032 [id=1322b48e2bf14cf2851b7f0e1828441c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.905262 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f77f8b613544e909ed3397c4b2ebd08. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:13.908720 21691 leader_election.cc:343] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:57:13.909166 21691 leader_election.cc:304] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
W20250112 02:57:13.910904 21691 raft_consensus.cc:2681] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P 1c2770701ca54288956df9d910d7d3d2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:57:13.913120 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:13.914817 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:13.920269 22153 ts_tablet_manager.cc:1918] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.920619 22153 log.cc:1198] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/3497ff26cfbe4a73b1f7bb54b83a32a7
I20250112 02:57:13.921267 22153 ts_tablet_manager.cc:1939] T 3497ff26cfbe4a73b1f7bb54b83a32a7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:13.923388 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 (DEFAULT_TABLE table=default.Table_419f8e3cdd474bdfbcead2ec83af6032 [id=1322b48e2bf14cf2851b7f0e1828441c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:13.924413 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f77f8b613544e909ed3397c4b2ebd08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:13.934006 22144 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:13.934041 22044 raft_consensus.cc:2463] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:13.935050 21385 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_60e29c461b3f4842aecb00c8d23d169f" } new_table_name: "default.Table_ed0ee963f50b43e39c034e36ebbacd4e" modify_external_catalogs: true
I20250112 02:57:13.939445 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 3497ff26cfbe4a73b1f7bb54b83a32a7 (table default.Table_97850d925a00442ab1e1296aa4c87e45 [id=e2c68fb77f964468905c2b7b699408b5]) successfully deleted
I20250112 02:57:13.942082 22145 tablet_bootstrap.cc:654] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.949239 22152 ts_tablet_manager.cc:1918] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:13.949803 22152 log.cc:1198] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/1fdb1b8b1db746abb082a1039ee9268b
I20250112 02:57:13.953509 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_dec32526aa174f35b386fb5ccc07c740" } modify_external_catalogs: true
I20250112 02:57:13.954066 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_dec32526aa174f35b386fb5ccc07c740" } modify_external_catalogs: true
W20250112 02:57:13.957074 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f77f8b613544e909ed3397c4b2ebd08 consensus state has no leader
I20250112 02:57:13.970963 22143 tablet_bootstrap.cc:654] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.977918 22144 tablet_bootstrap.cc:654] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:13.978191 22152 ts_tablet_manager.cc:1939] T 1fdb1b8b1db746abb082a1039ee9268b P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:13.981812 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 1fdb1b8b1db746abb082a1039ee9268b (table default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]) successfully deleted
I20250112 02:57:13.994745 22145 tablet_bootstrap.cc:492] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:13.995261 22145 ts_tablet_manager.cc:1397] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.321s user 0.008s sys 0.005s
I20250112 02:57:13.997735 22145 raft_consensus.cc:357] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:13.998379 22145 raft_consensus.cc:383] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:13.998718 22145 raft_consensus.cc:738] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.011600 22145 consensus_queue.cc:260] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.012529 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_475b0b23fef943389a87bc8e981d14c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.014233 22144 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
W20250112 02:57:14.014231 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_475b0b23fef943389a87bc8e981d14c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.018146 22144 ts_tablet_manager.cc:1397] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.003s
I20250112 02:57:14.020660 22144 raft_consensus.cc:357] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.021505 22144 raft_consensus.cc:383] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.021824 22144 raft_consensus.cc:738] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.022574 22144 consensus_queue.cc:260] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.024216 22145 ts_tablet_manager.cc:1428] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250112 02:57:14.024369 21391 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 2 got 0
I20250112 02:57:14.027480 22143 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:14.027863 22143 ts_tablet_manager.cc:1397] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.008s
I20250112 02:57:14.027819 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 15c16031f5b74cf9885fac110f445e1a (DEFAULT_TABLE table=default.Table_c41d4357625c47ccb556d7a2231c65fd [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.028822 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c16031f5b74cf9885fac110f445e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.030086 22144 ts_tablet_manager.cc:1428] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250112 02:57:14.030063 22143 raft_consensus.cc:357] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.030717 22143 raft_consensus.cc:383] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.030975 22143 raft_consensus.cc:738] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.031949 22143 consensus_queue.cc:260] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.036332 22145 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.036463 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 15c16031f5b74cf9885fac110f445e1a (DEFAULT_TABLE table=default.Table_c41d4357625c47ccb556d7a2231c65fd [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.037426 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c16031f5b74cf9885fac110f445e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.037807 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 15c16031f5b74cf9885fac110f445e1a (DEFAULT_TABLE table=default.Table_c41d4357625c47ccb556d7a2231c65fd [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.045099 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_38868f576fce4ebcb97f42b8f83724e8 [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 2 got 0
I20250112 02:57:14.050712 22145 tablet_bootstrap.cc:654] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.054030 22143 ts_tablet_manager.cc:1428] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
W20250112 02:57:14.064893 21385 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3b3af671bcea480a8f48b3c4abcc74ba consensus state has no leader
I20250112 02:57:14.065323 22145 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.065742 22145 ts_tablet_manager.cc:1397] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.001s
I20250112 02:57:14.066473 22143 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:14.068202 22145 raft_consensus.cc:357] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.068912 22145 raft_consensus.cc:383] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.069411 22145 raft_consensus.cc:738] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.070101 22145 consensus_queue.cc:260] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.075531 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_459b9af6b9cf4a20976238e01971d9fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.076853 21385 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_81e59853126a44968bc4c56e97efc6bc [id=32f1cf819ea748d78535c9c44d64b92e]). Expected version 1 got 0
W20250112 02:57:14.077116 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_459b9af6b9cf4a20976238e01971d9fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.078392 22145 ts_tablet_manager.cc:1428] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20250112 02:57:14.087019 22145 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.087208 22153 tablet_replica.cc:331] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:14.087901 22153 raft_consensus.cc:2238] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.088315 22153 raft_consensus.cc:2267] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.091794 22144 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:14.093315 21389 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_81e59853126a44968bc4c56e97efc6bc [id=32f1cf819ea748d78535c9c44d64b92e]). Expected version 1 got 0
I20250112 02:57:14.093530 22153 ts_tablet_manager.cc:1905] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.102698 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15c16031f5b74cf9885fac110f445e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.106786 22144 tablet_bootstrap.cc:654] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.114943 22143 tablet_bootstrap.cc:654] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.115192 22145 tablet_bootstrap.cc:654] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.124490 22153 ts_tablet_manager.cc:1918] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.125475 22153 log.cc:1198] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/1fdb1b8b1db746abb082a1039ee9268b
I20250112 02:57:14.126118 22153 ts_tablet_manager.cc:1939] T 1fdb1b8b1db746abb082a1039ee9268b P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:14.129093 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 1fdb1b8b1db746abb082a1039ee9268b (table default.Table_4c0ff8b26e54463c9f0c92213bd77228 [id=e00740b2398046dfa045e98caa35bfa4]) successfully deleted
I20250112 02:57:14.131389 22161 raft_consensus.cc:491] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.131834 22161 raft_consensus.cc:513] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.133714 22161 leader_election.cc:290] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:14.135109 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbebff35c344bdb305ea1a6dbbefe4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.135469 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbebff35c344bdb305ea1a6dbbefe4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:14.135776 21902 raft_consensus.cc:2463] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:14.136045 22044 raft_consensus.cc:2463] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:14.137101 21691 leader_election.cc:304] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:14.140003 22161 raft_consensus.cc:2798] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:14.140358 22161 raft_consensus.cc:491] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:14.140722 22161 raft_consensus.cc:3054] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.150348 22161 raft_consensus.cc:513] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.157320 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbebff35c344bdb305ea1a6dbbefe4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:14.157878 21902 raft_consensus.cc:3054] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.158771 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 06f24063285e4b32989514ab8863fb14
I20250112 02:57:14.162142 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.162793 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.163385 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.164758 22151 tablet_replica.cc:331] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:14.165001 21902 raft_consensus.cc:2463] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:14.165454 22151 raft_consensus.cc:2238] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.165894 22151 raft_consensus.cc:2267] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.166090 21691 leader_election.cc:304] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:14.166795 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 750dbf8e68cb457e8871641582c9ba27
W20250112 02:57:14.167351 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 06f24063285e4b32989514ab8863fb14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06f24063285e4b32989514ab8863fb14 already in progress: creating tablet
I20250112 02:57:14.170171 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 750dbf8e68cb457e8871641582c9ba27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.171109 22152 tablet_replica.cc:331] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:14.172137 22152 raft_consensus.cc:2238] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.172722 22152 raft_consensus.cc:2267] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.173746 22161 leader_election.cc:290] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:14.170037 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fbebff35c344bdb305ea1a6dbbefe4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:14.175197 22044 raft_consensus.cc:3054] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.174479 22161 raft_consensus.cc:2798] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:14.180459 22151 ts_tablet_manager.cc:1905] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.180941 22161 raft_consensus.cc:695] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:14.182356 22161 consensus_queue.cc:237] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.184070 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:14.184774 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 750dbf8e68cb457e8871641582c9ba27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:14.185477 22153 tablet_replica.cc:331] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:14.186193 22153 raft_consensus.cc:2238] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:14.186954 22153 pending_rounds.cc:62] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Trying to abort 1 pending ops.
I20250112 02:57:14.187208 22153 pending_rounds.cc:69] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113320996267126784 op_type: NO_OP noop_request { }
I20250112 02:57:14.187633 22153 raft_consensus.cc:2883] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:14.187976 22153 raft_consensus.cc:2267] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.188577 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a77c396c9dd148f4aea2dc0bce0ec411"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.190205 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a77c396c9dd148f4aea2dc0bce0ec411 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.190448 22153 ts_tablet_manager.cc:1905] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.191251 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_38868f576fce4ebcb97f42b8f83724e8" } new_table_name: "default.Table_dfa6a8556d1b4cbbad688309ba37fe5f" modify_external_catalogs: true
I20250112 02:57:14.198745 22152 ts_tablet_manager.cc:1905] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.203787 22144 tablet_bootstrap.cc:492] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:14.204159 22144 ts_tablet_manager.cc:1397] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.113s user 0.011s sys 0.005s
I20250112 02:57:14.207233 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 750dbf8e68cb457e8871641582c9ba27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.208050 22145 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.208469 22145 ts_tablet_manager.cc:1397] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.122s user 0.025s sys 0.005s
I20250112 02:57:14.210414 22145 raft_consensus.cc:357] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.211056 22145 raft_consensus.cc:383] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.211294 22145 raft_consensus.cc:738] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.206348 22144 raft_consensus.cc:357] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.211655 22144 raft_consensus.cc:383] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.211905 22144 raft_consensus.cc:738] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.211979 22145 consensus_queue.cc:260] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.212553 22144 consensus_queue.cc:260] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
W20250112 02:57:14.214391 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 06f24063285e4b32989514ab8863fb14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06f24063285e4b32989514ab8863fb14 already in progress: creating tablet
I20250112 02:57:14.208101 22044 raft_consensus.cc:2463] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:14.219297 22143 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:14.219707 22143 ts_tablet_manager.cc:1397] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.022s
I20250112 02:57:14.220995 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_d5caf55266494547a656e5afaf12bf4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.221942 22143 raft_consensus.cc:357] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.222608 22143 raft_consensus.cc:383] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.222880 22143 raft_consensus.cc:738] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.223882 22145 ts_tablet_manager.cc:1428] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
W20250112 02:57:14.223943 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5caf55266494547a656e5afaf12bf4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.232466 22145 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.242738 22144 ts_tablet_manager.cc:1428] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20250112 02:57:14.243643 22144 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:14.250732 22145 tablet_bootstrap.cc:654] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.250911 22143 consensus_queue.cc:260] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.251856 22197 raft_consensus.cc:491] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.252372 22197 raft_consensus.cc:513] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.256662 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faf24cad2f0456caa5a151a1a11a5a8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:14.257865 21690 leader_election.cc:343] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:14.258282 22153 ts_tablet_manager.cc:1918] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:14.258729 22153 log.cc:1198] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/750dbf8e68cb457e8871641582c9ba27
I20250112 02:57:14.259452 22153 ts_tablet_manager.cc:1939] T 750dbf8e68cb457e8871641582c9ba27 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:14.260347 22144 tablet_bootstrap.cc:654] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.260167 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faf24cad2f0456caa5a151a1a11a5a8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.260746 21902 raft_consensus.cc:2463] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:14.261889 21691 leader_election.cc:304] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters: e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:14.262621 22161 raft_consensus.cc:2798] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:14.262671 22197 leader_election.cc:290] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:14.262998 22161 raft_consensus.cc:491] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:14.263335 22161 raft_consensus.cc:3054] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.264838 22145 tablet_bootstrap.cc:492] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.265223 22145 ts_tablet_manager.cc:1397] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.002s
I20250112 02:57:14.267545 22145 raft_consensus.cc:357] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.267961 22151 ts_tablet_manager.cc:1918] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.268299 22151 log.cc:1198] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/06f24063285e4b32989514ab8863fb14
I20250112 02:57:14.269124 22151 ts_tablet_manager.cc:1939] T 06f24063285e4b32989514ab8863fb14 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:14.268183 22145 raft_consensus.cc:383] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.269482 22145 raft_consensus.cc:738] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.270893 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_be3f6ab8fc684025b5b231ca16c14c9e" } new_table_name: "default.Table_355d103eff5a46c2bd4aedc3e85aa88a" modify_external_catalogs: true
I20250112 02:57:14.272727 22145 consensus_queue.cc:260] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.273236 22143 ts_tablet_manager.cc:1428] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.053s user 0.005s sys 0.001s
I20250112 02:57:14.274040 22143 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:14.277233 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11fbebff35c344bdb305ea1a6dbbefe4 consensus state has no leader
I20250112 02:57:14.281773 21881 tablet_service.cc:1467] Processing CreateTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.282847 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bb62ab3f274c29b2c5f9528890b816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.283226 22023 tablet_service.cc:1467] Processing CreateTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.283488 22152 ts_tablet_manager.cc:1918] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.283728 22145 ts_tablet_manager.cc:1428] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250112 02:57:14.283876 22152 log.cc:1198] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/750dbf8e68cb457e8871641582c9ba27
I20250112 02:57:14.284236 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bb62ab3f274c29b2c5f9528890b816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.288856 22145 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.289844 21736 tablet_service.cc:1467] Processing CreateTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.290760 22151 tablet_replica.cc:331] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:14.290895 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9bb62ab3f274c29b2c5f9528890b816. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.291561 22151 raft_consensus.cc:2238] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.292021 22151 raft_consensus.cc:2267] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.292848 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_3cbd6af691c1497bb9829994c20e0645 [id=67a9c7c8457e4345ac77ab0e82bd682a]) successfully deleted
I20250112 02:57:14.295725 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 06f24063285e4b32989514ab8863fb14 (table default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]) successfully deleted
I20250112 02:57:14.295939 22143 tablet_bootstrap.cc:654] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:14.296432 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 06f24063285e4b32989514ab8863fb14 because the tablet was not found. No further retry: Not found: Tablet not found: 06f24063285e4b32989514ab8863fb14
I20250112 02:57:14.298982 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_81e59853126a44968bc4c56e97efc6bc [id=32f1cf819ea748d78535c9c44d64b92e]). Expected version 1 got 0
I20250112 02:57:14.299515 22152 ts_tablet_manager.cc:1939] T 750dbf8e68cb457e8871641582c9ba27 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:14.300901 22145 tablet_bootstrap.cc:654] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.302338 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_3cbd6af691c1497bb9829994c20e0645 [id=67a9c7c8457e4345ac77ab0e82bd682a]) successfully deleted
I20250112 02:57:14.310297 21878 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfe4c753f84aeeba7446def9898d43 (DEFAULT_TABLE table=default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.310954 22022 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfe4c753f84aeeba7446def9898d43 (DEFAULT_TABLE table=default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.311386 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfe4c753f84aeeba7446def9898d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.312215 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfe4c753f84aeeba7446def9898d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.315253 21733 tablet_service.cc:1467] Processing CreateTablet for tablet a6bfe4c753f84aeeba7446def9898d43 (DEFAULT_TABLE table=default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.316792 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6bfe4c753f84aeeba7446def9898d43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.318866 22145 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.319219 22145 ts_tablet_manager.cc:1397] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20250112 02:57:14.323490 22145 raft_consensus.cc:357] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.324119 22145 raft_consensus.cc:383] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.324359 22145 raft_consensus.cc:738] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.325035 22145 consensus_queue.cc:260] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.326126 22161 raft_consensus.cc:513] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.343060 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faf24cad2f0456caa5a151a1a11a5a8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:14.343621 22044 raft_consensus.cc:3054] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.344110 22145 ts_tablet_manager.cc:1428] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20250112 02:57:14.294380 22151 ts_tablet_manager.cc:1905] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.350633 22044 raft_consensus.cc:2463] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:14.351610 21690 leader_election.cc:304] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:14.353093 22145 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.352680 22199 raft_consensus.cc:2798] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:14.353650 22199 raft_consensus.cc:695] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:14.354355 22199 consensus_queue.cc:237] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.358150 22145 tablet_bootstrap.cc:654] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.367354 22163 raft_consensus.cc:491] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.367780 22163 raft_consensus.cc:513] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:14.369221 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f77f8b613544e909ed3397c4b2ebd08 consensus state has no leader
I20250112 02:57:14.369719 22163 leader_election.cc:290] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.371562 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3af671bcea480a8f48b3c4abcc74ba" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.372639 22143 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
W20250112 02:57:14.372901 21980 leader_election.cc:343] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:14.373152 22143 ts_tablet_manager.cc:1397] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.000s
I20250112 02:57:14.373620 22161 leader_election.cc:290] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:14.374058 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b3af671bcea480a8f48b3c4abcc74ba" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:14.376297 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6faf24cad2f0456caa5a151a1a11a5a8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:14.376847 21902 raft_consensus.cc:3054] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.377029 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ed0ee963f50b43e39c034e36ebbacd4e" } modify_external_catalogs: true
I20250112 02:57:14.377683 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ed0ee963f50b43e39c034e36ebbacd4e" } modify_external_catalogs: true
I20250112 02:57:14.380472 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_0162c6f72cb74aa9a0e65d704255d604" } modify_external_catalogs: true
W20250112 02:57:14.381093 21978 leader_election.cc:343] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:14.381336 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_0162c6f72cb74aa9a0e65d704255d604" } modify_external_catalogs: true
I20250112 02:57:14.381474 21978 leader_election.cc:304] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:14.382654 22163 raft_consensus.cc:2743] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:14.383617 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_610cf7d84db445aeab7898cdbaf84ab7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.385587 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_610cf7d84db445aeab7898cdbaf84ab7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.388000 21902 raft_consensus.cc:2463] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:14.392129 22145 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.392511 22145 ts_tablet_manager.cc:1397] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20250112 02:57:14.395782 22143 raft_consensus.cc:357] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.396629 22143 raft_consensus.cc:383] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.396983 22143 raft_consensus.cc:738] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.397184 22145 raft_consensus.cc:357] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.397814 22145 raft_consensus.cc:383] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.398046 22145 raft_consensus.cc:738] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.400080 22144 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:14.400456 22144 ts_tablet_manager.cc:1397] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.000s
I20250112 02:57:14.402652 22144 raft_consensus.cc:357] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.403287 22144 raft_consensus.cc:383] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.403539 22144 raft_consensus.cc:738] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.406867 22143 consensus_queue.cc:260] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.410753 22144 consensus_queue.cc:260] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.413776 22144 ts_tablet_manager.cc:1428] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250112 02:57:14.418076 22145 consensus_queue.cc:260] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.419449 22144 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:14.420910 22143 ts_tablet_manager.cc:1428] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250112 02:57:14.421878 22143 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:14.429963 22145 ts_tablet_manager.cc:1428] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20250112 02:57:14.437709 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a35bc1e625384c6da78f2547aaf31bec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.439402 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a35bc1e625384c6da78f2547aaf31bec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.443789 22145 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.450959 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a (DEFAULT_TABLE table=default.Table_459b9af6b9cf4a20976238e01971d9fd [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.452178 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a5c293bbd643aab9409c6e9abe4f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.451390 21385 catalog_manager.cc:5526] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:14.453626 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_dfa6a8556d1b4cbbad688309ba37fe5f [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 3 got 0
I20250112 02:57:14.455164 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 3b3af671bcea480a8f48b3c4abcc74ba (table default.Table_ed0ee963f50b43e39c034e36ebbacd4e [id=417cdf8adb1841e1a3793f3db4e112cc]). Expected version 1 got 0
I20250112 02:57:14.455530 22145 tablet_bootstrap.cc:654] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.459896 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a (DEFAULT_TABLE table=default.Table_459b9af6b9cf4a20976238e01971d9fd [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.462416 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (DEFAULT_TABLE table=default.Table_475b0b23fef943389a87bc8e981d14c9 [id=ed996933e39d472992d6f351ae51fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.462724 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (DEFAULT_TABLE table=default.Table_475b0b23fef943389a87bc8e981d14c9 [id=ed996933e39d472992d6f351ae51fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.463598 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f556cd7d6f4e13a525d3b8d4207d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.467415 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a5c293bbd643aab9409c6e9abe4f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.471659 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f556cd7d6f4e13a525d3b8d4207d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.491219 22145 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.491600 22145 ts_tablet_manager.cc:1397] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.009s
I20250112 02:57:14.498445 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_81a50ebd3f5c40dd9893310926f5439e" } new_table_name: "default.Table_1eb229f782f54f29a10fee46bc4785c1" modify_external_catalogs: true
I20250112 02:57:14.499323 22145 raft_consensus.cc:357] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.499970 22145 raft_consensus.cc:383] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.500214 22145 raft_consensus.cc:738] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.507304 22145 consensus_queue.cc:260] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.508046 22163 raft_consensus.cc:491] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.508164 22151 ts_tablet_manager.cc:1918] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.508654 22151 log.cc:1198] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/750dbf8e68cb457e8871641582c9ba27
I20250112 02:57:14.508424 22163 raft_consensus.cc:513] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.509550 22151 ts_tablet_manager.cc:1939] T 750dbf8e68cb457e8871641582c9ba27 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:14.512017 22143 tablet_bootstrap.cc:654] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.513844 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90643c97b5e424a8bf3c262e3e5429e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:14.515108 21978 leader_election.cc:343] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:14.515591 22163 leader_election.cc:290] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.519969 22145 ts_tablet_manager.cc:1428] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20250112 02:57:14.528103 22145 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.528035 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_50bcfd9aa201416e976e67c56aa7f3f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.531193 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b9cd8852182c4c8f8988154152c71bc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.533491 22145 tablet_bootstrap.cc:654] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:14.533442 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9cd8852182c4c8f8988154152c71bc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:14.530387 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50bcfd9aa201416e976e67c56aa7f3f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.545042 22163 raft_consensus.cc:491] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.545454 22163 raft_consensus.cc:513] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.547127 22163 leader_election.cc:290] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.548434 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737bb9fd0c324e0d8ba7e4bc373be80e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:14.548985 21756 raft_consensus.cc:2463] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:14.549883 21978 leader_election.cc:304] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:14.550602 22163 raft_consensus.cc:2798] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:14.550900 22163 raft_consensus.cc:491] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:14.551178 22163 raft_consensus.cc:3054] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.552045 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 3b3af671bcea480a8f48b3c4abcc74ba
I20250112 02:57:14.553611 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.556478 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.572912 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_459b9af6b9cf4a20976238e01971d9fd" } new_table_name: "default.Table_45510e3d77df4fa899940125a4a5da9c" modify_external_catalogs: true
I20250112 02:57:14.574268 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_3cbd6af691c1497bb9829994c20e0645 [id=67a9c7c8457e4345ac77ab0e82bd682a]) successfully deleted
I20250112 02:57:14.580667 22143 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:14.581198 22143 ts_tablet_manager.cc:1397] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.159s user 0.010s sys 0.000s
I20250112 02:57:14.584656 22143 raft_consensus.cc:357] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.585631 22143 raft_consensus.cc:383] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.585951 22143 raft_consensus.cc:738] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:14.587705 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 3b3af671bcea480a8f48b3c4abcc74ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b3af671bcea480a8f48b3c4abcc74ba already in progress: creating tablet
I20250112 02:57:14.588678 22144 tablet_bootstrap.cc:654] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.592592 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90643c97b5e424a8bf3c262e3e5429e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.593508 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737bb9fd0c324e0d8ba7e4bc373be80e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.595420 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a (DEFAULT_TABLE table=default.Table_459b9af6b9cf4a20976238e01971d9fd [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.596663 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33a5c293bbd643aab9409c6e9abe4f0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.598611 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:14.599695 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (DEFAULT_TABLE table=default.Table_475b0b23fef943389a87bc8e981d14c9 [id=ed996933e39d472992d6f351ae51fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.600801 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f556cd7d6f4e13a525d3b8d4207d94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.603130 21902 raft_consensus.cc:2463] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:14.609340 22143 consensus_queue.cc:260] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.621387 22143 ts_tablet_manager.cc:1428] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.040s user 0.009s sys 0.000s
I20250112 02:57:14.622332 22143 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:14.638144 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 3b3af671bcea480a8f48b3c4abcc74ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b3af671bcea480a8f48b3c4abcc74ba already in progress: creating tablet
I20250112 02:57:14.554785 22152 tablet_replica.cc:331] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:14.681032 22152 raft_consensus.cc:2238] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.681854 22152 raft_consensus.cc:2267] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.683975 22152 ts_tablet_manager.cc:1905] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.687114 22143 tablet_bootstrap.cc:654] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:14.691788 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 750dbf8e68cb457e8871641582c9ba27 (table default.Table_3cbd6af691c1497bb9829994c20e0645 [id=67a9c7c8457e4345ac77ab0e82bd682a], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 750dbf8e68cb457e8871641582c9ba27 consensus state has no leader
I20250112 02:57:14.701078 22144 tablet_bootstrap.cc:492] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:14.701689 22144 ts_tablet_manager.cc:1397] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.282s user 0.009s sys 0.007s
W20250112 02:57:14.713368 21980 leader_election.cc:343] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:14.714085 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 92683.7us
I20250112 02:57:14.714404 22210 raft_consensus.cc:491] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.714907 21980 leader_election.cc:304] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:14.715073 22210 raft_consensus.cc:513] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.718519 22163 raft_consensus.cc:513] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.721359 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737bb9fd0c324e0d8ba7e4bc373be80e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:14.721891 21756 raft_consensus.cc:3054] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.728250 21756 raft_consensus.cc:2463] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:14.733083 21384 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 6faf24cad2f0456caa5a151a1a11a5a8
I20250112 02:57:14.740173 22144 raft_consensus.cc:357] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.740844 22144 raft_consensus.cc:383] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.741091 22144 raft_consensus.cc:738] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.743160 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_432c0b6f219d4c11b784bd6913108e00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.744778 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_432c0b6f219d4c11b784bd6913108e00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.741626 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.747367 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e608afe71349f1ae609e584022e510" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:14.748260 21757 raft_consensus.cc:2463] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:14.748884 22151 tablet_replica.cc:331] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:14.748812 21978 leader_election.cc:304] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:14.749626 22151 raft_consensus.cc:2238] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:14.750524 22151 pending_rounds.cc:62] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Trying to abort 1 pending ops.
I20250112 02:57:14.750869 22151 pending_rounds.cc:69] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113320998326231040 op_type: NO_OP noop_request { }
I20250112 02:57:14.751438 22151 raft_consensus.cc:2883] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:14.751855 22151 raft_consensus.cc:2267] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.766371 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:14.767993 22210 leader_election.cc:290] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:14.770890 22153 tablet_replica.cc:331] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:14.771531 22210 raft_consensus.cc:2798] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:14.771817 22153 raft_consensus.cc:2238] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.772249 22210 raft_consensus.cc:491] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:14.772369 22153 raft_consensus.cc:2267] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.772820 22210 raft_consensus.cc:3054] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.774294 22153 ts_tablet_manager.cc:1905] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.774224 22214 raft_consensus.cc:2798] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:14.774721 22214 raft_consensus.cc:695] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:14.775755 22214 consensus_queue.cc:237] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.781585 22144 consensus_queue.cc:260] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.784276 21978 leader_election.cc:304] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:14.788367 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 73571.6us
I20250112 02:57:14.790400 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "737bb9fd0c324e0d8ba7e4bc373be80e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:14.790974 21902 raft_consensus.cc:3054] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.795540 22144 ts_tablet_manager.cc:1428] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.093s user 0.003s sys 0.002s
I20250112 02:57:14.796880 21902 raft_consensus.cc:2463] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:14.797461 22151 ts_tablet_manager.cc:1905] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.799377 22211 raft_consensus.cc:2798] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:14.799721 22211 raft_consensus.cc:491] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:14.800024 22211 raft_consensus.cc:3054] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.802525 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e608afe71349f1ae609e584022e510" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.803032 21902 raft_consensus.cc:2463] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:14.806041 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 6faf24cad2f0456caa5a151a1a11a5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.806655 22163 leader_election.cc:290] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.809902 21387 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_355d103eff5a46c2bd4aedc3e85aa88a" } modify_external_catalogs: true
I20250112 02:57:14.810400 21387 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_355d103eff5a46c2bd4aedc3e85aa88a" } modify_external_catalogs: true
I20250112 02:57:14.812698 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b264bbff78dd43b7ac519d9d1b26e30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.814827 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b264bbff78dd43b7ac519d9d1b26e30a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:14.821190 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a5c293bbd643aab9409c6e9abe4f0a consensus state has no leader
I20250112 02:57:14.817768 22143 tablet_bootstrap.cc:492] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:14.823475 22143 ts_tablet_manager.cc:1397] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.201s user 0.017s sys 0.002s
I20250112 02:57:14.823997 21902 raft_consensus.cc:1270] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c2770701ca54288956df9d910d7d3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:57:14.825109 22044 raft_consensus.cc:1270] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Refusing update from remote peer 1c2770701ca54288956df9d910d7d3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:57:14.825512 22199 consensus_queue.cc:1035] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:14.826822 22161 consensus_queue.cc:1035] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:14.827695 22144 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:14.837939 22145 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:14.827630 22143 raft_consensus.cc:357] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.842059 22152 ts_tablet_manager.cc:1918] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.842494 22143 raft_consensus.cc:383] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:14.842118 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 57942.5us
I20250112 02:57:14.842878 22143 raft_consensus.cc:738] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.844027 22143 consensus_queue.cc:260] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.851256 22145 ts_tablet_manager.cc:1397] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.323s user 0.007s sys 0.017s
I20250112 02:57:14.852548 22143 ts_tablet_manager.cc:1428] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250112 02:57:14.867551 22143 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:14.875082 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a90643c97b5e424a8bf3c262e3e5429e consensus state has no leader
I20250112 02:57:14.876535 21733 tablet_service.cc:1467] Processing CreateTablet for tablet cad224c5ad844512b50da5190cfd1861 (DEFAULT_TABLE table=default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.878057 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad224c5ad844512b50da5190cfd1861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.881747 22145 raft_consensus.cc:357] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.882268 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 (DEFAULT_TABLE table=default.Table_a77c396c9dd148f4aea2dc0bce0ec411 [id=e084f29dae5a41899d7cb805c1ea63b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.884466 21882 tablet_service.cc:1467] Processing CreateTablet for tablet cad224c5ad844512b50da5190cfd1861 (DEFAULT_TABLE table=default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.885581 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.862152 22151 ts_tablet_manager.cc:1918] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:14.887567 22151 log.cc:1198] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/6faf24cad2f0456caa5a151a1a11a5a8
I20250112 02:57:14.882441 22145 raft_consensus.cc:383] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.888489 22151 ts_tablet_manager.cc:1939] T 6faf24cad2f0456caa5a151a1a11a5a8 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:14.888067 22145 raft_consensus.cc:738] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.890559 22145 consensus_queue.cc:260] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.893913 22144 tablet_bootstrap.cc:654] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.903632 22143 tablet_bootstrap.cc:654] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:14.903106 22145 ts_tablet_manager.cc:1428] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.051s user 0.006s sys 0.002s
I20250112 02:57:14.904940 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 (DEFAULT_TABLE table=default.Table_a77c396c9dd148f4aea2dc0bce0ec411 [id=e084f29dae5a41899d7cb805c1ea63b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.906030 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f787ffe65e34e8f9852d149fff9f947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.907864 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 6faf24cad2f0456caa5a151a1a11a5a8 (table default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]) successfully deleted
I20250112 02:57:14.908125 22022 tablet_service.cc:1467] Processing CreateTablet for tablet cad224c5ad844512b50da5190cfd1861 (DEFAULT_TABLE table=default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.842415 22152 log.cc:1198] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/3b3af671bcea480a8f48b3c4abcc74ba
I20250112 02:57:14.909133 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad224c5ad844512b50da5190cfd1861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.909574 22163 raft_consensus.cc:491] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:14.909727 22152 ts_tablet_manager.cc:1939] T 3b3af671bcea480a8f48b3c4abcc74ba P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:14.909972 22163 raft_consensus.cc:513] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.912084 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 (DEFAULT_TABLE table=default.Table_a77c396c9dd148f4aea2dc0bce0ec411 [id=e084f29dae5a41899d7cb805c1ea63b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:14.913055 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f787ffe65e34e8f9852d149fff9f947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.913646 22151 tablet_replica.cc:331] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:14.914398 22151 raft_consensus.cc:2238] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:14.916280 22151 raft_consensus.cc:2267] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:14.918740 21388 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_dfa6a8556d1b4cbbad688309ba37fe5f [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 3 got 0
I20250112 02:57:14.925516 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f787ffe65e34e8f9852d149fff9f947. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.925854 22210 raft_consensus.cc:513] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:14.927625 22210 leader_election.cc:290] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.929948 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.915318 22224 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:14.930640 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90643c97b5e424a8bf3c262e3e5429e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:14.931252 21900 raft_consensus.cc:3054] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.931334 22163 leader_election.cc:290] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:14.934828 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90643c97b5e424a8bf3c262e3e5429e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:14.930630 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06f24063285e4b32989514ab8863fb14" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:14.936868 21756 raft_consensus.cc:3054] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:14.937775 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 1f77f8b613544e909ed3397c4b2ebd08
W20250112 02:57:14.938297 21978 leader_election.cc:343] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 06f24063285e4b32989514ab8863fb14
I20250112 02:57:14.940196 22151 ts_tablet_manager.cc:1905] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:14.940335 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:14.942418 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:14.943439 22145 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:14.950999 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cad224c5ad844512b50da5190cfd1861. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:14.953543 22220 mvcc.cc:204] Tried to move back new op lower bound from 7113321000218558464 to 7113320997734739968. Current Snapshot: MvccSnapshot[applied={T|T < 7113321000218558464}]
I20250112 02:57:14.954618 22153 ts_tablet_manager.cc:1918] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:14.954967 22153 log.cc:1198] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/6faf24cad2f0456caa5a151a1a11a5a8
I20250112 02:57:14.957362 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:14.958005 22153 ts_tablet_manager.cc:1939] T 6faf24cad2f0456caa5a151a1a11a5a8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:14.964926 22143 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:14.965456 22143 ts_tablet_manager.cc:1397] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.002s
I20250112 02:57:14.966977 22219 mvcc.cc:204] Tried to move back new op lower bound from 7113321000218558464 to 7113320997734739968. Current Snapshot: MvccSnapshot[applied={T|T < 7113321000218558464}]
W20250112 02:57:14.958021 21980 leader_election.cc:343] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:14.967661 21980 leader_election.cc:304] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:14.968751 22217 raft_consensus.cc:2743] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:14.970175 22143 raft_consensus.cc:357] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.973593 22143 raft_consensus.cc:383] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:14.974588 22143 raft_consensus.cc:738] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:14.975284 22143 consensus_queue.cc:260] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:14.990761 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e8c3cd631d684a6b9d725c23343be273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:14.991986 22143 ts_tablet_manager.cc:1428] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
W20250112 02:57:14.992549 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8c3cd631d684a6b9d725c23343be273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.992764 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_fffe2a1efd5e4ff3973a842a73b42291"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:14.994340 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fffe2a1efd5e4ff3973a842a73b42291 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:14.997273 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 6faf24cad2f0456caa5a151a1a11a5a8 (table default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]) successfully deleted
I20250112 02:57:14.997609 22211 raft_consensus.cc:513] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
W20250112 02:57:14.999032 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 1f77f8b613544e909ed3397c4b2ebd08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f77f8b613544e909ed3397c4b2ebd08 already in progress: creating tablet
I20250112 02:57:15.000778 22211 leader_election.cc:290] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:15.001014 21900 raft_consensus.cc:2463] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:15.001449 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e608afe71349f1ae609e584022e510" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:15.001951 21757 raft_consensus.cc:3054] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.002133 21756 raft_consensus.cc:2463] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:15.003181 21978 leader_election.cc:304] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.003404 22152 tablet_replica.cc:331] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:15.004293 22152 raft_consensus.cc:2238] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.004722 22152 raft_consensus.cc:2267] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.005561 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 3b3af671bcea480a8f48b3c4abcc74ba (table default.Table_ed0ee963f50b43e39c034e36ebbacd4e [id=417cdf8adb1841e1a3793f3db4e112cc]) successfully deleted
I20250112 02:57:15.006446 22152 ts_tablet_manager.cc:1905] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.007485 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e608afe71349f1ae609e584022e510" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:15.007954 22211 raft_consensus.cc:2798] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:15.008014 21900 raft_consensus.cc:3054] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.008448 22211 raft_consensus.cc:695] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:15.009418 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b9cd8852182c4c8f8988154152c71bc7" } modify_external_catalogs: true
I20250112 02:57:15.009213 22211 consensus_queue.cc:237] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.009943 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b9cd8852182c4c8f8988154152c71bc7" } modify_external_catalogs: true
I20250112 02:57:15.011741 22143 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:15.017575 22143 tablet_bootstrap.cc:654] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.026844 21390 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_dfa6a8556d1b4cbbad688309ba37fe5f [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 3 got 0
I20250112 02:57:15.028478 22145 tablet_bootstrap.cc:654] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.032423 22227 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:15.036944 22228 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:15.045074 21390 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_dfa6a8556d1b4cbbad688309ba37fe5f" } new_table_name: "default.Table_04002b2285af49eb9d1fd9edeb48810c" modify_external_catalogs: true
I20250112 02:57:15.077008 22143 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:15.077409 22143 ts_tablet_manager.cc:1397] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.013s
I20250112 02:57:15.079478 22144 tablet_bootstrap.cc:492] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:15.079834 22144 ts_tablet_manager.cc:1397] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.252s user 0.009s sys 0.003s
I20250112 02:57:15.079614 22143 raft_consensus.cc:357] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.080205 22143 raft_consensus.cc:383] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.080438 22143 raft_consensus.cc:738] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.080994 22143 consensus_queue.cc:260] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.083273 22144 raft_consensus.cc:357] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.083905 22144 raft_consensus.cc:383] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.084188 22144 raft_consensus.cc:738] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.088538 21757 raft_consensus.cc:2463] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:15.088541 21900 raft_consensus.cc:2463] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:15.089493 22151 ts_tablet_manager.cc:1918] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.089852 22151 log.cc:1198] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/3b3af671bcea480a8f48b3c4abcc74ba
I20250112 02:57:15.090468 22151 ts_tablet_manager.cc:1939] T 3b3af671bcea480a8f48b3c4abcc74ba P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:15.092797 21978 leader_election.cc:304] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.093554 22211 raft_consensus.cc:2798] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:15.094729 22211 raft_consensus.cc:695] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:15.097437 22211 consensus_queue.cc:237] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.102319 21384 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 4 got 3
I20250112 02:57:15.107156 22145 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:15.107172 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250112 02:57:15.107683 22145 ts_tablet_manager.cc:1397] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.008s
I20250112 02:57:15.109911 22145 raft_consensus.cc:357] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.110731 21387 catalog_manager.cc:5526] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:15.111617 22144 consensus_queue.cc:260] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.112406 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 4 got 0
I20250112 02:57:15.113401 22143 ts_tablet_manager.cc:1428] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250112 02:57:15.118990 21389 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_1eb229f782f54f29a10fee46bc4785c1 [id=14a6d904311440469400b09be94d6748]). Expected version 1 got 0
I20250112 02:57:15.121348 22151 tablet_replica.cc:331] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:15.118876 22145 raft_consensus.cc:383] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.122937 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 3b3af671bcea480a8f48b3c4abcc74ba (table default.Table_ed0ee963f50b43e39c034e36ebbacd4e [id=417cdf8adb1841e1a3793f3db4e112cc]) successfully deleted
I20250112 02:57:15.123553 22145 raft_consensus.cc:738] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.131166 22151 raft_consensus.cc:2238] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.131680 22151 raft_consensus.cc:2267] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.132977 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.133613 22144 ts_tablet_manager.cc:1428] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.054s user 0.006s sys 0.000s
I20250112 02:57:15.135815 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.138406 22143 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:15.130254 22145 consensus_queue.cc:260] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.138819 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.140532 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.141930 22144 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:15.146338 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de9ed3d1552490da7ab76037d7debc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.151265 22144 tablet_bootstrap.cc:654] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.157384 22145 ts_tablet_manager.cc:1428] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s
I20250112 02:57:15.157281 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 7d699356dd3341338670b433a7fa02ad (DEFAULT_TABLE table=default.Table_a35bc1e625384c6da78f2547aaf31bec [id=8aa9ef226db340358d3a91700aa33602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.157469 21390 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a35bc1e625384c6da78f2547aaf31bec" } new_table_name: "default.Table_f2f1590c7a40439c89ff74db3d19cdcf" modify_external_catalogs: true
I20250112 02:57:15.162473 22145 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.151790 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301c8a2c534a4c54bfd6e678f3184ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.165866 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d699356dd3341338670b433a7fa02ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.172894 22143 tablet_bootstrap.cc:654] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.182888 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 7d699356dd3341338670b433a7fa02ad (DEFAULT_TABLE table=default.Table_a35bc1e625384c6da78f2547aaf31bec [id=8aa9ef226db340358d3a91700aa33602]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.171986 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.174297 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.187299 22145 tablet_bootstrap.cc:654] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.188604 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_86735ff40c83478eafeb3d073a86f588" } modify_external_catalogs: true
I20250112 02:57:15.189097 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_86735ff40c83478eafeb3d073a86f588" } modify_external_catalogs: true
I20250112 02:57:15.189322 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de9ed3d1552490da7ab76037d7debc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.189531 22218 mvcc.cc:204] Tried to move back new op lower bound from 7113321000218558464 to 7113320997734739968. Current Snapshot: MvccSnapshot[applied={T|T < 7113321000218558464}]
I20250112 02:57:15.190583 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301c8a2c534a4c54bfd6e678f3184ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.191843 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_68a1e71fb38145f797bf5a1740443cb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.193465 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68a1e71fb38145f797bf5a1740443cb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.197335 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d699356dd3341338670b433a7fa02ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.198029 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_c18efd91f8804ad69b7532dbba118163"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:15.200335 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 7d699356dd3341338670b433a7fa02ad (DEFAULT_TABLE table=default.Table_a35bc1e625384c6da78f2547aaf31bec [id=8aa9ef226db340358d3a91700aa33602]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:15.201164 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c18efd91f8804ad69b7532dbba118163 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.217423 22224 tablet.cc:1716] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:57:15.220623 22228 tablet.cc:1716] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:57:15.225011 22227 tablet.cc:1716] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Skipping requested alter to schema version 3, tablet already version 3
I20250112 02:57:15.231684 22145 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.232059 22145 ts_tablet_manager.cc:1397] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.006s
I20250112 02:57:15.234149 22145 raft_consensus.cc:357] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.238533 22145 raft_consensus.cc:383] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.238790 22145 raft_consensus.cc:738] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.239472 22145 consensus_queue.cc:260] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.239920 22151 ts_tablet_manager.cc:1905] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.244546 22227 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:57:15.248164 22145 ts_tablet_manager.cc:1428] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250112 02:57:15.248975 22145 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.253859 22228 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:57:15.258070 22224 tablet.cc:1721] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250112 02:57:15.262250 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_7f26edfb6fc44a20b7e131ccbabf6f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.263895 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.254387 22145 tablet_bootstrap.cc:654] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.267957 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b3af671bcea480a8f48b3c4abcc74ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:15.268656 22153 tablet_replica.cc:331] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:15.269376 22153 raft_consensus.cc:2238] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.269778 22153 raft_consensus.cc:2267] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.271930 22153 ts_tablet_manager.cc:1905] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.281070 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet bb324cd3120e45449e0500c99338135d
I20250112 02:57:15.280979 21737 tablet_service.cc:1467] Processing CreateTablet for tablet bb324cd3120e45449e0500c99338135d (DEFAULT_TABLE table=default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.282327 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb324cd3120e45449e0500c99338135d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.283658 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:15.284842 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:15.286640 21881 tablet_service.cc:1467] Processing CreateTablet for tablet bb324cd3120e45449e0500c99338135d (DEFAULT_TABLE table=default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.287592 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb324cd3120e45449e0500c99338135d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.298764 22152 ts_tablet_manager.cc:1918] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.299199 22152 log.cc:1198] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/6faf24cad2f0456caa5a151a1a11a5a8
I20250112 02:57:15.299858 22152 ts_tablet_manager.cc:1939] T 6faf24cad2f0456caa5a151a1a11a5a8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:15.301537 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d699356dd3341338670b433a7fa02ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.302969 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7de9ed3d1552490da7ab76037d7debc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.303881 22021 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:15.303934 22024 tablet_service.cc:1467] Processing CreateTablet for tablet bb324cd3120e45449e0500c99338135d (DEFAULT_TABLE table=default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.305229 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 301c8a2c534a4c54bfd6e678f3184ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.319327 21389 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]). Expected version 4 got 3
I20250112 02:57:15.329314 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_81e59853126a44968bc4c56e97efc6bc" } new_table_name: "default.Table_3827b9f8c5ad4db7865b690a90b4e7d4" modify_external_catalogs: true
I20250112 02:57:15.335573 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb324cd3120e45449e0500c99338135d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.339581 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ae37aca456fc4eb9a0dd51f067f4b34c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.341876 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae37aca456fc4eb9a0dd51f067f4b34c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:15.351230 21390 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d699356dd3341338670b433a7fa02ad consensus state has no leader
I20250112 02:57:15.388059 22145 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.388444 22145 ts_tablet_manager.cc:1397] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.140s user 0.032s sys 0.021s
I20250112 02:57:15.390442 22145 raft_consensus.cc:357] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.391075 22145 raft_consensus.cc:383] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.391306 22145 raft_consensus.cc:738] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.391903 22145 consensus_queue.cc:260] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.398461 22145 ts_tablet_manager.cc:1428] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250112 02:57:15.399238 22145 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.400693 22152 tablet_replica.cc:331] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:15.401484 22152 raft_consensus.cc:2238] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.401875 22152 raft_consensus.cc:2267] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.404560 22145 tablet_bootstrap.cc:654] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.409188 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 6faf24cad2f0456caa5a151a1a11a5a8 (table default.Table_0162c6f72cb74aa9a0e65d704255d604 [id=f6329d8614164120be0026c6efcd1ed7]) successfully deleted
I20250112 02:57:15.423874 22152 ts_tablet_manager.cc:1905] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.437790 22145 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.438184 22145 ts_tablet_manager.cc:1397] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.011s
I20250112 02:57:15.461743 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 737bb9fd0c324e0d8ba7e4bc373be80e
I20250112 02:57:15.464017 22143 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:15.464916 22143 ts_tablet_manager.cc:1397] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.327s user 0.027s sys 0.022s
I20250112 02:57:15.466765 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:15.467312 22145 raft_consensus.cc:357] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.467954 22145 raft_consensus.cc:383] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.468173 22145 raft_consensus.cc:738] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.472236 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:15.471560 22143 raft_consensus.cc:357] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.472368 22145 consensus_queue.cc:260] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.472973 22143 raft_consensus.cc:383] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.473234 22143 raft_consensus.cc:738] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.473245 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 737bb9fd0c324e0d8ba7e4bc373be80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:15.473800 22143 consensus_queue.cc:260] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.475983 22145 ts_tablet_manager.cc:1428] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20250112 02:57:15.476774 22145 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.481750 22143 ts_tablet_manager.cc:1428] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250112 02:57:15.482710 22143 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:15.489264 22145 tablet_bootstrap.cc:654] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.490197 22143 tablet_bootstrap.cc:654] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.512862 22145 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.513350 22145 ts_tablet_manager.cc:1397] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.008s
I20250112 02:57:15.515522 22145 raft_consensus.cc:357] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.516180 22145 raft_consensus.cc:383] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.516448 22145 raft_consensus.cc:738] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.517048 22145 consensus_queue.cc:260] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.519421 22145 ts_tablet_manager.cc:1428] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 02:57:15.521337 22145 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.523798 22143 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:15.524184 22143 ts_tablet_manager.cc:1397] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.000s
I20250112 02:57:15.527658 22144 tablet_bootstrap.cc:492] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:15.528069 22144 ts_tablet_manager.cc:1397] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.386s user 0.006s sys 0.005s
I20250112 02:57:15.530211 22144 raft_consensus.cc:357] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.530819 22144 raft_consensus.cc:383] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.531057 22144 raft_consensus.cc:738] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.531682 22144 consensus_queue.cc:260] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.533996 22144 ts_tablet_manager.cc:1428] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20250112 02:57:15.534829 22144 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:15.537662 22226 raft_consensus.cc:491] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:15.538138 22226 raft_consensus.cc:513] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.540063 22226 leader_election.cc:290] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:15.541988 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:15.542608 22045 raft_consensus.cc:2463] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:15.543570 22145 tablet_bootstrap.cc:654] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.543993 21690 leader_election.cc:304] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.548035 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_d5caf55266494547a656e5afaf12bf4b" } modify_external_catalogs: true
I20250112 02:57:15.548588 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_d5caf55266494547a656e5afaf12bf4b" } modify_external_catalogs: true
I20250112 02:57:15.552644 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:15.556706 21388 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1e608afe71349f1ae609e584022e510 consensus state has no leader
W20250112 02:57:15.556573 21691 leader_election.cc:343] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:15.559504 22226 raft_consensus.cc:2798] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:15.559885 22226 raft_consensus.cc:491] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:15.560263 22226 raft_consensus.cc:3054] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.560590 22145 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.560950 22145 ts_tablet_manager.cc:1397] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.011s
I20250112 02:57:15.563062 22145 raft_consensus.cc:357] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.563702 22145 raft_consensus.cc:383] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.563964 22145 raft_consensus.cc:738] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.564507 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_475b0b23fef943389a87bc8e981d14c9" } new_table_name: "default.Table_8f7412e5cea04b119aed9eab7eb1e8f9" modify_external_catalogs: true
I20250112 02:57:15.564564 22145 consensus_queue.cc:260] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.568256 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e268c8cc3ff74be4b1af3dd65ff78d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.569890 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e268c8cc3ff74be4b1af3dd65ff78d43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.573184 22145 ts_tablet_manager.cc:1428] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250112 02:57:15.573973 22145 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.579308 22144 tablet_bootstrap.cc:654] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.526198 22143 raft_consensus.cc:357] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.582032 22143 raft_consensus.cc:383] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.582285 22143 raft_consensus.cc:738] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.582890 22143 consensus_queue.cc:260] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.591804 22145 tablet_bootstrap.cc:654] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.594189 21383 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c] alter complete (version 4)
I20250112 02:57:15.598006 22153 ts_tablet_manager.cc:1918] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.598322 22151 ts_tablet_manager.cc:1918] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.598676 22151 log.cc:1198] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:15.598392 22153 log.cc:1198] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/3b3af671bcea480a8f48b3c4abcc74ba
I20250112 02:57:15.599336 22151 ts_tablet_manager.cc:1939] T 1f77f8b613544e909ed3397c4b2ebd08 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:15.599838 22153 ts_tablet_manager.cc:1939] T 3b3af671bcea480a8f48b3c4abcc74ba P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:15.600528 22226 raft_consensus.cc:513] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.602466 22226 leader_election.cc:290] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:15.606812 22152 ts_tablet_manager.cc:1918] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.607184 22152 log.cc:1198] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/06f24063285e4b32989514ab8863fb14
I20250112 02:57:15.607790 22152 ts_tablet_manager.cc:1939] T 06f24063285e4b32989514ab8863fb14 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:15.611832 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:15.612526 22045 raft_consensus.cc:3054] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.615561 22143 ts_tablet_manager.cc:1428] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.091s user 0.005s sys 0.001s
I20250112 02:57:15.620692 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:15.624301 21691 leader_election.cc:343] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:15.639178 22143 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:15.644236 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_8972a68b9140407885f2933d6362b3b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.645830 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8972a68b9140407885f2933d6362b3b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.648038 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 06f24063285e4b32989514ab8863fb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:13 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:15.654793 22045 raft_consensus.cc:2463] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:15.654928 22151 tablet_replica.cc:331] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:15.655607 22151 raft_consensus.cc:2238] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.656049 22151 raft_consensus.cc:2267] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.657647 22151 ts_tablet_manager.cc:1905] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.661374 22153 tablet_replica.cc:331] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:15.661288 21690 leader_election.cc:304] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:15.662115 22153 raft_consensus.cc:2238] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.662344 22226 raft_consensus.cc:2798] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:15.662760 22226 raft_consensus.cc:695] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:15.663545 22226 consensus_queue.cc:237] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.664245 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 3b3af671bcea480a8f48b3c4abcc74ba (table default.Table_ed0ee963f50b43e39c034e36ebbacd4e [id=417cdf8adb1841e1a3793f3db4e112cc]) successfully deleted
I20250112 02:57:15.664510 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 1f77f8b613544e909ed3397c4b2ebd08 (table default.Table_355d103eff5a46c2bd4aedc3e85aa88a [id=1322b48e2bf14cf2851b7f0e1828441c]) successfully deleted
W20250112 02:57:15.664815 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet bb324cd3120e45449e0500c99338135d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb324cd3120e45449e0500c99338135d already in progress: creating tablet
W20250112 02:57:15.665160 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet bb324cd3120e45449e0500c99338135d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb324cd3120e45449e0500c99338135d already in progress: creating tablet
I20250112 02:57:15.665714 22153 raft_consensus.cc:2267] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.667366 22153 ts_tablet_manager.cc:1905] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.669224 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 06f24063285e4b32989514ab8863fb14 (table default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]) successfully deleted
I20250112 02:57:15.672888 22152 tablet_replica.cc:331] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:15.673463 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3827b9f8c5ad4db7865b690a90b4e7d4" } modify_external_catalogs: true
I20250112 02:57:15.674111 22144 tablet_bootstrap.cc:492] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:15.673991 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3827b9f8c5ad4db7865b690a90b4e7d4" } modify_external_catalogs: true
I20250112 02:57:15.674465 22144 ts_tablet_manager.cc:1397] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.140s user 0.007s sys 0.000s
I20250112 02:57:15.675302 22152 raft_consensus.cc:2238] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.675765 22152 raft_consensus.cc:2267] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.677424 22152 ts_tablet_manager.cc:1905] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.677198 22144 raft_consensus.cc:357] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.677783 22144 raft_consensus.cc:383] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.678143 22144 raft_consensus.cc:738] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.678857 22144 consensus_queue.cc:260] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.681586 22143 tablet_bootstrap.cc:654] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:15.696718 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7d699356dd3341338670b433a7fa02ad consensus state has no leader
I20250112 02:57:15.699052 22144 ts_tablet_manager.cc:1428] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20250112 02:57:15.699827 22144 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:15.712397 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 3781431324bc479d97bd22b6bb1243b9 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.713500 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3781431324bc479d97bd22b6bb1243b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.713827 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 3781431324bc479d97bd22b6bb1243b9 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.714895 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3781431324bc479d97bd22b6bb1243b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.715325 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e (DEFAULT_TABLE table=default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.718266 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e (DEFAULT_TABLE table=default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.719290 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5890ebb3c55943d5a334d6c93fbdaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.719866 22144 tablet_bootstrap.cc:654] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.722160 22153 ts_tablet_manager.cc:1918] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.722559 22153 log.cc:1198] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/737bb9fd0c324e0d8ba7e4bc373be80e
I20250112 02:57:15.723168 22153 ts_tablet_manager.cc:1939] T 737bb9fd0c324e0d8ba7e4bc373be80e P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:15.723289 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e (DEFAULT_TABLE table=default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.724308 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5890ebb3c55943d5a334d6c93fbdaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.725610 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 737bb9fd0c324e0d8ba7e4bc373be80e (table default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]) successfully deleted
I20250112 02:57:15.726778 22153 tablet_replica.cc:331] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:15.727450 22153 raft_consensus.cc:2238] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:15.718292 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 3781431324bc479d97bd22b6bb1243b9 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.728896 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3781431324bc479d97bd22b6bb1243b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.729566 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5890ebb3c55943d5a334d6c93fbdaf0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.729926 22144 tablet_bootstrap.cc:492] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:15.730283 22144 ts_tablet_manager.cc:1397] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.007s
I20250112 02:57:15.732553 22144 raft_consensus.cc:357] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.733227 22144 raft_consensus.cc:383] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.733477 22144 raft_consensus.cc:738] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.734354 22151 ts_tablet_manager.cc:1918] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.734702 22151 log.cc:1198] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/737bb9fd0c324e0d8ba7e4bc373be80e
I20250112 02:57:15.734642 22211 raft_consensus.cc:491] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:15.735299 22151 ts_tablet_manager.cc:1939] T 737bb9fd0c324e0d8ba7e4bc373be80e P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:15.735049 22211 raft_consensus.cc:513] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:15.736729 21798 debug-util.cc:398] Leaking SignalData structure 0x7b08000eda20 after lost signal to thread 21632
I20250112 02:57:15.736882 22211 leader_election.cc:290] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:15.737995 21798 debug-util.cc:398] Leaking SignalData structure 0x7b08000edd80 after lost signal to thread 21801
I20250112 02:57:15.738567 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c16031f5b74cf9885fac110f445e1a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:15.738866 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c16031f5b74cf9885fac110f445e1a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:15.740255 21757 raft_consensus.cc:2463] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:15.741385 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 737bb9fd0c324e0d8ba7e4bc373be80e (table default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]) successfully deleted
I20250112 02:57:15.741389 21978 leader_election.cc:304] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.742175 22211 raft_consensus.cc:2798] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:15.742514 22211 raft_consensus.cc:491] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:15.742867 22211 raft_consensus.cc:3054] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.742409 22144 consensus_queue.cc:260] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.743589 21900 raft_consensus.cc:2463] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:15.751106 22144 ts_tablet_manager.cc:1428] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20250112 02:57:15.754281 22144 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:15.759871 22144 tablet_bootstrap.cc:654] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.761197 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.765707 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.766973 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402c24e54ab040e29949b1e4efbe5ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.766773 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.767887 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402c24e54ab040e29949b1e4efbe5ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.770812 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 402c24e54ab040e29949b1e4efbe5ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.771816 22144 tablet_bootstrap.cc:492] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:15.772235 22144 ts_tablet_manager.cc:1397] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.002s
I20250112 02:57:15.774343 22144 raft_consensus.cc:357] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.775295 22144 raft_consensus.cc:383] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.775578 22144 raft_consensus.cc:738] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.776243 22144 consensus_queue.cc:260] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.781227 22144 ts_tablet_manager.cc:1428] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250112 02:57:15.784396 22144 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:15.785830 22145 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.786213 22145 ts_tablet_manager.cc:1397] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.212s user 0.019s sys 0.031s
I20250112 02:57:15.788430 22145 raft_consensus.cc:357] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.789044 22145 raft_consensus.cc:383] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.789271 22145 raft_consensus.cc:738] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.790122 22145 consensus_queue.cc:260] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.796232 22144 tablet_bootstrap.cc:654] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.796409 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 27234f3e69d544bdb53fb887874c2c0a (DEFAULT_TABLE table=default.Table_432c0b6f219d4c11b784bd6913108e00 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.797483 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27234f3e69d544bdb53fb887874c2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.801016 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 27234f3e69d544bdb53fb887874c2c0a (DEFAULT_TABLE table=default.Table_432c0b6f219d4c11b784bd6913108e00 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.802063 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27234f3e69d544bdb53fb887874c2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.804683 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 27234f3e69d544bdb53fb887874c2c0a (DEFAULT_TABLE table=default.Table_432c0b6f219d4c11b784bd6913108e00 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.805706 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27234f3e69d544bdb53fb887874c2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.805904 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 43c413e1f8fe48039c4dd1b066fdd857 (DEFAULT_TABLE table=default.Table_b264bbff78dd43b7ac519d9d1b26e30a [id=cd412b57888e46bba00cfdffd6cfa143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.806929 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c413e1f8fe48039c4dd1b066fdd857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.807157 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 43c413e1f8fe48039c4dd1b066fdd857 (DEFAULT_TABLE table=default.Table_b264bbff78dd43b7ac519d9d1b26e30a [id=cd412b57888e46bba00cfdffd6cfa143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.808733 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c413e1f8fe48039c4dd1b066fdd857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.821076 22145 ts_tablet_manager.cc:1428] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.035s user 0.008s sys 0.004s
W20250112 02:57:15.822196 21978 consensus_peers.cc:487] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 737bb9fd0c324e0d8ba7e4bc373be80e. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:15.836866 22143 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:15.837284 22143 ts_tablet_manager.cc:1397] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.198s user 0.002s sys 0.006s
I20250112 02:57:15.838721 22145 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.839404 22143 raft_consensus.cc:357] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.840063 22143 raft_consensus.cc:383] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.840322 22143 raft_consensus.cc:738] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.840863 22143 consensus_queue.cc:260] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.856945 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_1eb229f782f54f29a10fee46bc4785c1" } new_table_name: "default.Table_f77d231547584322affd733f7e6103ba" modify_external_catalogs: true
I20250112 02:57:15.861557 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_882a7ca6ecaf41179e63ff807c02a0dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:15.863370 22143 ts_tablet_manager.cc:1428] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
W20250112 02:57:15.863451 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_882a7ca6ecaf41179e63ff807c02a0dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.869261 22143 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:15.881908 22143 tablet_bootstrap.cc:654] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.886812 22145 tablet_bootstrap.cc:654] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.889048 22152 ts_tablet_manager.cc:1918] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:15.889367 22152 log.cc:1198] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:15.889278 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 43c413e1f8fe48039c4dd1b066fdd857 (DEFAULT_TABLE table=default.Table_b264bbff78dd43b7ac519d9d1b26e30a [id=cd412b57888e46bba00cfdffd6cfa143]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:15.890265 22152 ts_tablet_manager.cc:1939] T 1f77f8b613544e909ed3397c4b2ebd08 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:15.890381 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43c413e1f8fe48039c4dd1b066fdd857. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:15.892287 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 1f77f8b613544e909ed3397c4b2ebd08 (table default.Table_355d103eff5a46c2bd4aedc3e85aa88a [id=1322b48e2bf14cf2851b7f0e1828441c]) successfully deleted
I20250112 02:57:15.888119 22250 raft_consensus.cc:491] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:15.834082 22211 raft_consensus.cc:513] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:15.894975 22250 raft_consensus.cc:513] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.897118 22250 leader_election.cc:290] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:15.897820 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:15.898420 21757 raft_consensus.cc:2463] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:15.900028 21978 leader_election.cc:304] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.901907 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c16031f5b74cf9885fac110f445e1a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:15.902470 21757 raft_consensus.cc:3054] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:15.903538 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet bb324cd3120e45449e0500c99338135d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb324cd3120e45449e0500c99338135d already in progress: creating tablet
I20250112 02:57:15.904398 22152 tablet_replica.cc:331] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:15.905105 22152 raft_consensus.cc:2238] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:15.905737 22152 pending_rounds.cc:62] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Trying to abort 1 pending ops.
I20250112 02:57:15.906018 22152 pending_rounds.cc:69] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321000481177600 op_type: NO_OP noop_request { }
I20250112 02:57:15.906384 22152 raft_consensus.cc:2883] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:15.906697 22152 raft_consensus.cc:2267] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.906666 22250 raft_consensus.cc:2798] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:15.907068 22250 raft_consensus.cc:491] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:15.907357 22250 raft_consensus.cc:3054] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.909245 21757 raft_consensus.cc:2463] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:15.910348 21978 leader_election.cc:304] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:15.912842 22152 ts_tablet_manager.cc:1905] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.920411 22145 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:15.920753 22145 ts_tablet_manager.cc:1397] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.000s
I20250112 02:57:15.922721 22145 raft_consensus.cc:357] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.923658 22145 raft_consensus.cc:383] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.923897 22145 raft_consensus.cc:738] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.924052 22248 raft_consensus.cc:2798] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:15.924472 22145 consensus_queue.cc:260] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.901360 22211 leader_election.cc:290] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:15.934878 22145 ts_tablet_manager.cc:1428] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250112 02:57:15.935667 22145 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:15.919006 22250 raft_consensus.cc:513] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:15.941939 22250 leader_election.cc:290] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:15.953141 22145 tablet_bootstrap.cc:654] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:15.953953 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:15.954612 21757 raft_consensus.cc:3054] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.727882 22153 raft_consensus.cc:2267] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:15.960563 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a15b58bd8c49483082451ae033d064b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:15.962182 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a15b58bd8c49483082451ae033d064b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:15.959296 22154 raft_consensus.cc:491] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:15.966640 22153 ts_tablet_manager.cc:1905] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:15.970710 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:15.972913 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c16031f5b74cf9885fac110f445e1a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:15.973594 21900 raft_consensus.cc:3054] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:15.980437 22144 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
W20250112 02:57:15.980527 21980 leader_election.cc:343] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:15.980811 22144 ts_tablet_manager.cc:1397] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.197s user 0.022s sys 0.036s
I20250112 02:57:15.983474 22144 raft_consensus.cc:357] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.984124 22144 raft_consensus.cc:383] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:15.984364 22144 raft_consensus.cc:738] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:15.985101 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:15.986315 21980 leader_election.cc:343] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:15.986972 22144 consensus_queue.cc:260] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.968605 22154 raft_consensus.cc:513] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:15.991454 22143 tablet_bootstrap.cc:492] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:15.991817 22143 ts_tablet_manager.cc:1397] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.123s user 0.013s sys 0.006s
I20250112 02:57:15.993008 22154 leader_election.cc:290] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:15.994897 22152 ts_tablet_manager.cc:1918] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:15.995241 22152 log.cc:1198] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/737bb9fd0c324e0d8ba7e4bc373be80e
I20250112 02:57:15.995967 22152 ts_tablet_manager.cc:1939] T 737bb9fd0c324e0d8ba7e4bc373be80e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:15.997704 22248 raft_consensus.cc:695] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:15.998268 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 737bb9fd0c324e0d8ba7e4bc373be80e (table default.Table_86735ff40c83478eafeb3d073a86f588 [id=989f9c1d6ede40409cddd8562b3ed171]) successfully deleted
I20250112 02:57:15.998651 22248 consensus_queue.cc:237] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:16.001545 22250 raft_consensus.cc:491] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.001924 22250 raft_consensus.cc:513] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.002195 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.003602 22153 ts_tablet_manager.cc:1918] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:16.003599 22250 leader_election.cc:290] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.003938 22153 log.cc:1198] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/06f24063285e4b32989514ab8863fb14
I20250112 02:57:16.004072 22250 consensus_queue.cc:1035] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:16.004544 22153 ts_tablet_manager.cc:1939] T 06f24063285e4b32989514ab8863fb14 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:16.006664 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 06f24063285e4b32989514ab8863fb14 (table default.Table_dec32526aa174f35b386fb5ccc07c740 [id=874130907a9c48f1b9a5ac9543aeba79]) successfully deleted
I20250112 02:57:16.006963 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad224c5ad844512b50da5190cfd1861" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.008460 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad224c5ad844512b50da5190cfd1861" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.008935 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6bfe4c753f84aeeba7446def9898d43" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:16.009682 21980 leader_election.cc:343] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.010141 22045 raft_consensus.cc:2388] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:16.012068 21836 leader_election.cc:304] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:15.999488 22143 raft_consensus.cc:357] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.014993 22143 raft_consensus.cc:383] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.015283 22143 raft_consensus.cc:738] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.017388 22144 ts_tablet_manager.cc:1428] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.036s user 0.010s sys 0.010s
I20250112 02:57:16.020255 22259 raft_consensus.cc:3054] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.020778 22144 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:16.011322 21978 leader_election.cc:343] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.022847 22143 consensus_queue.cc:260] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.026805 21978 leader_election.cc:304] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.032670 22226 raft_consensus.cc:491] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.033042 22226 raft_consensus.cc:513] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.034801 22226 leader_election.cc:290] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.036280 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f556cd7d6f4e13a525d3b8d4207d94" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.036860 22045 raft_consensus.cc:2463] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:16.037930 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f556cd7d6f4e13a525d3b8d4207d94" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.038542 21690 leader_election.cc:304] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.039259 22226 raft_consensus.cc:2798] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:57:16.039463 21691 leader_election.cc:343] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.039535 22226 raft_consensus.cc:491] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.039817 22226 raft_consensus.cc:3054] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.044696 22143 ts_tablet_manager.cc:1428] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.053s user 0.005s sys 0.001s
I20250112 02:57:16.049901 22144 tablet_bootstrap.cc:654] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.051254 21900 raft_consensus.cc:2463] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.056213 22143 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.056612 22256 raft_consensus.cc:2743] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.059265 22144 tablet_bootstrap.cc:492] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.059643 22144 ts_tablet_manager.cc:1397] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.006s
I20250112 02:57:16.061616 22144 raft_consensus.cc:357] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.062215 22144 raft_consensus.cc:383] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.062487 22144 raft_consensus.cc:738] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.063071 22144 consensus_queue.cc:260] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.065151 22144 ts_tablet_manager.cc:1428] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250112 02:57:16.065898 22144 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.049649 22226 raft_consensus.cc:513] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:16.066006 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 54044.7us
I20250112 02:57:16.076864 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:36049
--webserver_interface=127.20.217.189
--webserver_port=45653
--builtin_ntp_servers=127.20.217.148:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:57:16.079404 22161 raft_consensus.cc:491] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.079823 22161 raft_consensus.cc:513] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.084367 22259 raft_consensus.cc:2743] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.089083 22145 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.089526 22145 ts_tablet_manager.cc:1397] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.154s user 0.037s sys 0.029s
I20250112 02:57:16.095758 22144 tablet_bootstrap.cc:654] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.106352 22144 tablet_bootstrap.cc:492] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.106786 22144 ts_tablet_manager.cc:1397] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.006s
I20250112 02:57:16.108762 22144 raft_consensus.cc:357] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.109524 22144 raft_consensus.cc:383] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.109802 22144 raft_consensus.cc:738] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.110461 22144 consensus_queue.cc:260] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.112919 22144 ts_tablet_manager.cc:1428] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250112 02:57:16.114249 22144 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.117439 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet cad224c5ad844512b50da5190cfd1861
I20250112 02:57:16.119557 22144 tablet_bootstrap.cc:654] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.123183 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:16.128793 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet cad224c5ad844512b50da5190cfd1861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cad224c5ad844512b50da5190cfd1861 already in progress: creating tablet
I20250112 02:57:16.131664 22144 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.132030 22144 ts_tablet_manager.cc:1397] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20250112 02:57:16.134003 22144 raft_consensus.cc:357] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.134639 22144 raft_consensus.cc:383] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.135740 22145 raft_consensus.cc:357] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.136385 22145 raft_consensus.cc:383] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.136618 22145 raft_consensus.cc:738] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.137161 22145 consensus_queue.cc:260] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.139099 22257 raft_consensus.cc:491] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.139473 22257 raft_consensus.cc:513] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.141213 22257 leader_election.cc:290] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.144620 22226 leader_election.cc:290] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.146663 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:16.147732 22161 leader_election.cc:290] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.148953 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f556cd7d6f4e13a525d3b8d4207d94" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:16.149209 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.149497 22045 raft_consensus.cc:3054] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.149744 22044 raft_consensus.cc:2463] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:16.150003 22143 tablet_bootstrap.cc:654] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.150197 22145 ts_tablet_manager.cc:1428] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20250112 02:57:16.150950 22152 tablet_replica.cc:331] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:16.151001 22145 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:16.151634 22152 raft_consensus.cc:2238] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:16.152031 22152 raft_consensus.cc:2267] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.151984 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_c43f3390842845678abdfbf57e7ff752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:16.152452 22256 raft_consensus.cc:491] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.152848 22256 raft_consensus.cc:513] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.146445 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.153786 21902 raft_consensus.cc:2463] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:16.155159 21980 leader_election.cc:304] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.156558 22257 raft_consensus.cc:2798] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.156862 22257 raft_consensus.cc:491] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.157209 22257 raft_consensus.cc:3054] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:16.153602 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c43f3390842845678abdfbf57e7ff752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.164901 22257 raft_consensus.cc:513] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.167279 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f787ffe65e34e8f9852d149fff9f947" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.167935 22256 leader_election.cc:290] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.134879 22144 raft_consensus.cc:738] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.172044 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.172662 21756 raft_consensus.cc:2463] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:16.173089 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f787ffe65e34e8f9852d149fff9f947" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.173486 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.174062 21901 raft_consensus.cc:2463] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:16.169946 22144 consensus_queue.cc:260] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.175144 21902 raft_consensus.cc:2463] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:16.175791 21691 leader_election.cc:304] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:16.176633 21980 leader_election.cc:304] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:16.177196 21978 leader_election.cc:343] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.178375 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:16.182651 22144 ts_tablet_manager.cc:1428] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.050s user 0.006s sys 0.001s
I20250112 02:57:16.183528 22144 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.186715 22045 raft_consensus.cc:2463] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:16.189152 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.189687 21902 raft_consensus.cc:3054] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.192603 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f556cd7d6f4e13a525d3b8d4207d94" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.193261 21900 raft_consensus.cc:3054] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.188723 22248 raft_consensus.cc:2798] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.194808 22248 raft_consensus.cc:491] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.195150 22248 raft_consensus.cc:3054] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.194909 22256 consensus_queue.cc:1035] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:16.185878 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:16.189632 21690 leader_election.cc:304] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.197235 22226 raft_consensus.cc:2798] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.190922 22161 raft_consensus.cc:2798] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.199008 22161 raft_consensus.cc:491] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.199338 22161 raft_consensus.cc:3054] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.201599 21902 raft_consensus.cc:2463] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.202925 21980 leader_election.cc:304] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.195276 22152 ts_tablet_manager.cc:1905] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:16.206996 22211 raft_consensus.cc:2798] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.215240 22145 tablet_bootstrap.cc:654] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.214315 21900 raft_consensus.cc:2463] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:16.222292 22256 consensus_queue.cc:1035] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:16.223794 22143 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.224279 22143 ts_tablet_manager.cc:1397] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.168s user 0.007s sys 0.009s
I20250112 02:57:16.228009 22248 raft_consensus.cc:513] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.228790 22143 raft_consensus.cc:357] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.229492 22143 raft_consensus.cc:383] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.229743 22143 raft_consensus.cc:738] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.230417 22211 raft_consensus.cc:695] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
W20250112 02:57:16.231324 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet cad224c5ad844512b50da5190cfd1861 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cad224c5ad844512b50da5190cfd1861 already in progress: creating tablet
I20250112 02:57:16.231468 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f787ffe65e34e8f9852d149fff9f947" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:16.231184 22211 consensus_queue.cc:237] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
W20250112 02:57:16.232640 21978 leader_election.cc:343] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.234723 22257 leader_election.cc:290] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.173085 21757 raft_consensus.cc:2463] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.238556 22257 consensus_queue.cc:1035] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:16.242796 22143 consensus_queue.cc:260] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.244501 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f787ffe65e34e8f9852d149fff9f947" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.245026 21902 raft_consensus.cc:3054] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.245894 22143 ts_tablet_manager.cc:1428] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20250112 02:57:16.253084 21902 raft_consensus.cc:2463] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.254149 21980 leader_election.cc:304] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:16.255148 22257 raft_consensus.cc:2798] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.255635 22257 raft_consensus.cc:695] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:16.256408 22257 consensus_queue.cc:237] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.258477 22143 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.258311 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_5ffe090e2d664579985eb72894c11d10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:16.258810 22248 leader_election.cc:290] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
W20250112 02:57:16.259976 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ffe090e2d664579985eb72894c11d10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.261914 22144 tablet_bootstrap.cc:654] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.282078 22143 tablet_bootstrap.cc:654] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.292593 22145 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.293054 22145 ts_tablet_manager.cc:1397] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.142s user 0.014s sys 0.004s
I20250112 02:57:16.294612 21383 catalog_manager.cc:5526] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:16.296011 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_1eb229f782f54f29a10fee46bc4785c1 [id=14a6d904311440469400b09be94d6748]). Expected version 1 got 0
I20250112 02:57:16.297111 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:16.298707 21384 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet b1e608afe71349f1ae609e584022e510
I20250112 02:57:16.301602 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e608afe71349f1ae609e584022e510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:16.302006 22144 tablet_bootstrap.cc:492] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.302352 22144 ts_tablet_manager.cc:1397] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.119s user 0.008s sys 0.007s
I20250112 02:57:16.303551 22143 tablet_bootstrap.cc:492] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.303946 22143 ts_tablet_manager.cc:1397] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.003s
W20250112 02:57:16.304363 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 12f556cd7d6f4e13a525d3b8d4207d94 consensus state has no leader
I20250112 02:57:16.304529 22144 raft_consensus.cc:357] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.305153 22144 raft_consensus.cc:383] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.305382 22144 raft_consensus.cc:738] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.306128 22143 raft_consensus.cc:357] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.306778 22143 raft_consensus.cc:383] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.307051 22143 raft_consensus.cc:738] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.308825 22144 consensus_queue.cc:260] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.302853 22152 ts_tablet_manager.cc:1918] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:16.310390 22152 log.cc:1198] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/cad224c5ad844512b50da5190cfd1861
I20250112 02:57:16.310515 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e608afe71349f1ae609e584022e510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:16.311173 22153 tablet_replica.cc:331] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:16.311820 22153 raft_consensus.cc:2238] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:16.312181 22144 ts_tablet_manager.cc:1428] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.010s user 0.004s sys 0.001s
I20250112 02:57:16.312260 22153 raft_consensus.cc:2267] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.312914 22144 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.313570 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e608afe71349f1ae609e584022e510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:16.314280 22151 tablet_replica.cc:331] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:16.314472 22153 ts_tablet_manager.cc:1905] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:16.316040 21978 leader_election.cc:304] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.316761 22257 raft_consensus.cc:2798] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.317188 22257 raft_consensus.cc:695] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:16.316901 22145 raft_consensus.cc:357] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.317544 22145 raft_consensus.cc:383] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.318150 22274 raft_consensus.cc:491] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.317960 22257 consensus_queue.cc:237] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.318970 22152 ts_tablet_manager.cc:1939] T cad224c5ad844512b50da5190cfd1861 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:16.318570 22274 raft_consensus.cc:513] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.320480 22274 leader_election.cc:290] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.323004 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad224c5ad844512b50da5190cfd1861" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.323058 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad224c5ad844512b50da5190cfd1861" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.323657 21902 raft_consensus.cc:2463] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:16.323915 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet cad224c5ad844512b50da5190cfd1861 (table default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]) successfully deleted
W20250112 02:57:16.325625 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a90643c97b5e424a8bf3c262e3e5429e consensus state has no leader
I20250112 02:57:16.325711 22152 tablet_replica.cc:331] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:16.317818 22145 raft_consensus.cc:738] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.326588 21691 leader_election.cc:304] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:16.326849 22143 consensus_queue.cc:260] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:16.327414 21690 leader_election.cc:343] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: Tablet not RUNNING: SHUTDOWN
I20250112 02:57:16.328891 22274 raft_consensus.cc:2798] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.329223 22274 raft_consensus.cc:491] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.329519 22274 raft_consensus.cc:3054] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.327175 22145 consensus_queue.cc:260] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.336545 22145 ts_tablet_manager.cc:1428] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
I20250112 02:57:16.339113 22274 raft_consensus.cc:513] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.340042 22145 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:16.341862 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cad224c5ad844512b50da5190cfd1861" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.328891 22143 ts_tablet_manager.cc:1428] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20250112 02:57:16.342381 21902 raft_consensus.cc:3054] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.348335 22143 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.349040 22161 raft_consensus.cc:513] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.349638 21902 raft_consensus.cc:2463] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:16.350246 21756 raft_consensus.cc:2388] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0406c08e9714e87bdd7fd6b66c9756d in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:16.350791 21691 leader_election.cc:304] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:16.355369 22153 ts_tablet_manager.cc:1918] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:16.355707 22153 log.cc:1198] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/b1e608afe71349f1ae609e584022e510
I20250112 02:57:16.356389 22153 ts_tablet_manager.cc:1939] T b1e608afe71349f1ae609e584022e510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:16.358103 22145 tablet_bootstrap.cc:654] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.360970 22144 tablet_bootstrap.cc:654] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.364600 22143 tablet_bootstrap.cc:654] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.364816 22259 raft_consensus.cc:491] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.365224 22259 raft_consensus.cc:513] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.367029 22259 leader_election.cc:290] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.368842 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.371007 22152 raft_consensus.cc:2238] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:16.373863 22152 raft_consensus.cc:2267] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.375366 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet b1e608afe71349f1ae609e584022e510 (table default.Table_3827b9f8c5ad4db7865b690a90b4e7d4 [id=32f1cf819ea748d78535c9c44d64b92e]) successfully deleted
I20250112 02:57:16.375274 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f6351a54c2544bfe94a9b4c219f69ad5" } modify_external_catalogs: true
I20250112 02:57:16.375833 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f6351a54c2544bfe94a9b4c219f69ad5" } modify_external_catalogs: true
I20250112 02:57:16.380658 22152 ts_tablet_manager.cc:1905] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:16.381073 22273 raft_consensus.cc:2798] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.381961 22273 raft_consensus.cc:695] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:16.384205 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a5c293bbd643aab9409c6e9abe4f0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.385632 21756 raft_consensus.cc:2388] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:16.387073 21835 leader_election.cc:304] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:16.388180 22259 raft_consensus.cc:3054] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.385536 22273 consensus_queue.cc:237] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.395633 22259 raft_consensus.cc:2743] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.404577 22145 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.405062 22145 ts_tablet_manager.cc:1397] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.002s
I20250112 02:57:16.402998 22226 raft_consensus.cc:695] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:16.409387 22145 raft_consensus.cc:357] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.414920 22145 raft_consensus.cc:383] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.415259 22145 raft_consensus.cc:738] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.411480 22226 consensus_queue.cc:237] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.417178 22145 consensus_queue.cc:260] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.419275 22144 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.419652 22144 ts_tablet_manager.cc:1397] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.009s
I20250112 02:57:16.421730 22144 raft_consensus.cc:357] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.422362 22144 raft_consensus.cc:383] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.422610 22144 raft_consensus.cc:738] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.425446 22274 leader_election.cc:290] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:16.426728 21690 leader_election.cc:343] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: cad224c5ad844512b50da5190cfd1861
I20250112 02:57:16.427436 22151 raft_consensus.cc:2238] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:16.430033 22151 pending_rounds.cc:62] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Trying to abort 1 pending ops.
I20250112 02:57:16.430258 22151 pending_rounds.cc:69] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321001418571776 op_type: NO_OP noop_request { }
I20250112 02:57:16.430817 22151 raft_consensus.cc:2883] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:16.430963 22145 ts_tablet_manager.cc:1428] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.026s user 0.006s sys 0.003s
I20250112 02:57:16.431123 22151 raft_consensus.cc:2267] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.430903 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:16.434656 22144 consensus_queue.cc:260] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.437954 22144 ts_tablet_manager.cc:1428] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 02:57:16.443740 22277 consensus_queue.cc:1035] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:16.445483 22145 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:16.451900 22152 ts_tablet_manager.cc:1918] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:16.452394 22152 log.cc:1198] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/b1e608afe71349f1ae609e584022e510
I20250112 02:57:16.453249 22152 ts_tablet_manager.cc:1939] T b1e608afe71349f1ae609e584022e510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:16.454713 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet b1e608afe71349f1ae609e584022e510 (table default.Table_3827b9f8c5ad4db7865b690a90b4e7d4 [id=32f1cf819ea748d78535c9c44d64b92e]) successfully deleted
I20250112 02:57:16.465687 22143 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.466192 22143 ts_tablet_manager.cc:1397] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.010s
I20250112 02:57:16.468039 22145 tablet_bootstrap.cc:654] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.468379 22143 raft_consensus.cc:357] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.471115 22143 raft_consensus.cc:383] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.471503 22143 raft_consensus.cc:738] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.473251 22143 consensus_queue.cc:260] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.477658 22144 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.483469 22144 tablet_bootstrap.cc:654] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.487888 22143 ts_tablet_manager.cc:1428] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250112 02:57:16.488771 22143 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.494613 22145 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.495045 22145 ts_tablet_manager.cc:1397] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.001s
W20250112 02:57:16.496062 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 26) took 798 ms (client timeout 1000 ms). Trace:
W20250112 02:57:16.496444 21389 rpcz_store.cc:269] 0112 02:57:15.697756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:15.699289 (+ 1533us) service_pool.cc:225] Handling call
0112 02:57:15.855338 (+156049us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:16.496001 (+640663us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":356461440}
I20250112 02:57:16.498355 22145 raft_consensus.cc:357] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.505784 22145 raft_consensus.cc:383] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.506074 22145 raft_consensus.cc:738] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.508397 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e8c3cd631d684a6b9d725c23343be273" } modify_external_catalogs: true
I20250112 02:57:16.509049 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e8c3cd631d684a6b9d725c23343be273" } modify_external_catalogs: true
I20250112 02:57:16.509132 22145 consensus_queue.cc:260] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.514756 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e268c8cc3ff74be4b1af3dd65ff78d43" } new_table_name: "default.Table_d17ad1e3dae4401f9292ff4a782c02e3" modify_external_catalogs: true
I20250112 02:57:16.521915 22145 ts_tablet_manager.cc:1428] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250112 02:57:16.524672 22143 tablet_bootstrap.cc:654] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.525303 22154 raft_consensus.cc:491] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.525339 22144 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.525861 22154 raft_consensus.cc:513] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } }
I20250112 02:57:16.526363 22144 ts_tablet_manager.cc:1397] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.008s
I20250112 02:57:16.527698 22154 leader_election.cc:290] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:16.528836 22145 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:16.532496 21835 leader_election.cc:343] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:16.533504 22144 raft_consensus.cc:357] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.534183 22144 raft_consensus.cc:383] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.534384 22144 raft_consensus.cc:738] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
W20250112 02:57:16.536334 21836 leader_election.cc:343] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:16.536247 22144 consensus_queue.cc:260] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.536725 21836 leader_election.cc:304] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:16.537381 22154 raft_consensus.cc:2743] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.540026 22144 ts_tablet_manager.cc:1428] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250112 02:57:16.540742 22144 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.545661 22299 tablet.cc:1721] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:16.548955 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_882a7ca6ecaf41179e63ff807c02a0dc" } new_table_name: "default.Table_52150679084b4cd39148ad19903eebcd" modify_external_catalogs: true
I20250112 02:57:16.552711 22145 tablet_bootstrap.cc:654] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.560817 22301 tablet.cc:1721] T a90643c97b5e424a8bf3c262e3e5429e P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:16.562119 22144 tablet_bootstrap.cc:654] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.562678 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:16.563867 22153 tablet_replica.cc:331] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:16.564566 22153 raft_consensus.cc:2238] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:16.565016 22153 raft_consensus.cc:2267] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.567018 22153 ts_tablet_manager.cc:1905] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:16.571820 22259 raft_consensus.cc:491] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.572459 22259 raft_consensus.cc:513] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.579910 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.579903 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.580508 22045 raft_consensus.cc:2463] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
W20250112 02:57:16.581179 21835 leader_election.cc:343] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.581626 21836 leader_election.cc:304] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:16.582511 22154 raft_consensus.cc:2798] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.582830 22154 raft_consensus.cc:491] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.583101 22154 raft_consensus.cc:3054] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.588059 22145 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.588508 22145 ts_tablet_manager.cc:1397] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.004s
I20250112 02:57:16.588651 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a560e6a044654680a00c3488dc51ce17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:16.590250 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a560e6a044654680a00c3488dc51ce17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.591328 22145 raft_consensus.cc:357] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.592175 22145 raft_consensus.cc:383] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.592496 22145 raft_consensus.cc:738] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.602170 22154 raft_consensus.cc:513] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.616199 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_1ab713178ff6472aa38bb55cd1c55d5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:16.618299 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ab713178ff6472aa38bb55cd1c55d5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.604602 22145 consensus_queue.cc:260] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.623091 22257 raft_consensus.cc:491] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.623601 22257 raft_consensus.cc:513] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.625293 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_84ec605ca7bd4c48b8fc9e590652654c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:16.626972 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84ec605ca7bd4c48b8fc9e590652654c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.627992 22257 leader_election.cc:290] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.631083 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.631346 22145 ts_tablet_manager.cc:1428] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.043s user 0.008s sys 0.000s
W20250112 02:57:16.632442 21978 leader_election.cc:343] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.639415 22145 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:16.639333 22161 leader_election.cc:290] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.627789 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.639730 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9bb62ab3f274c29b2c5f9528890b816" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.640468 21900 raft_consensus.cc:2388] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
W20250112 02:57:16.641003 21980 leader_election.cc:343] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.641386 21980 leader_election.cc:304] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.642262 22257 raft_consensus.cc:2743] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.642593 22259 leader_election.cc:290] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.642628 21691 leader_election.cc:304] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:16.644116 22161 raft_consensus.cc:2743] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.646003 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
W20250112 02:57:16.647413 21835 leader_election.cc:343] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.656713 22145 tablet_bootstrap.cc:654] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.670706 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:16.671272 22045 raft_consensus.cc:3054] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.679878 22045 raft_consensus.cc:2463] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:16.681170 21836 leader_election.cc:304] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:16.682216 22259 raft_consensus.cc:2798] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.683214 22259 raft_consensus.cc:695] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:16.686266 22259 consensus_queue.cc:237] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.688784 22154 leader_election.cc:290] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:16.698222 22300 tablet.cc:1721] T a90643c97b5e424a8bf3c262e3e5429e P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:16.709970 22257 raft_consensus.cc:491] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.710371 22257 raft_consensus.cc:513] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.712818 22257 leader_election.cc:290] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:16.714838 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.715615 22151 ts_tablet_manager.cc:1905] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:16.716322 21980 leader_election.cc:343] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.722851 22145 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.723241 22145 ts_tablet_manager.cc:1397] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.000s
I20250112 02:57:16.725065 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.725730 21755 raft_consensus.cc:2463] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:16.726769 21978 leader_election.cc:304] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.727800 22257 raft_consensus.cc:2798] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.728262 22257 raft_consensus.cc:491] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.728602 22257 raft_consensus.cc:3054] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.735567 22145 raft_consensus.cc:357] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.736253 22145 raft_consensus.cc:383] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.736500 22145 raft_consensus.cc:738] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.737032 22145 consensus_queue.cc:260] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.749799 22257 raft_consensus.cc:513] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.759449 22143 tablet_bootstrap.cc:492] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.759858 22143 ts_tablet_manager.cc:1397] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.271s user 0.022s sys 0.016s
I20250112 02:57:16.760794 22273 consensus_queue.cc:1035] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:16.762158 22143 raft_consensus.cc:357] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.762794 22143 raft_consensus.cc:383] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.763036 22143 raft_consensus.cc:738] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:16.763100 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b1e608afe71349f1ae609e584022e510 (table default.Table_3827b9f8c5ad4db7865b690a90b4e7d4 [id=32f1cf819ea748d78535c9c44d64b92e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet b1e608afe71349f1ae609e584022e510 consensus state has no leader
I20250112 02:57:16.766798 22143 consensus_queue.cc:260] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.767263 22250 raft_consensus.cc:491] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.770706 22250 raft_consensus.cc:513] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.772495 22250 leader_election.cc:290] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:16.773422 22143 ts_tablet_manager.cc:1428] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250112 02:57:16.774098 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.774538 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.775172 21902 raft_consensus.cc:2463] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:16.776198 21980 leader_election.cc:304] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.777529 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:16.778734 21980 leader_election.cc:343] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.780697 22257 leader_election.cc:290] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:16.778928 22143 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.780920 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:16.781427 22250 raft_consensus.cc:2798] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:16.782764 21756 raft_consensus.cc:3054] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.782967 22250 raft_consensus.cc:491] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:16.783272 22250 raft_consensus.cc:3054] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.767760 22145 ts_tablet_manager.cc:1428] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
W20250112 02:57:16.782282 21978 leader_election.cc:343] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.787379 22145 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:16.790987 22250 raft_consensus.cc:513] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.793432 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:16.794102 21900 raft_consensus.cc:3054] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.795964 21756 raft_consensus.cc:2463] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.797205 21978 leader_election.cc:304] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.798115 22276 raft_consensus.cc:2798] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.798579 22276 raft_consensus.cc:695] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:16.799630 22250 leader_election.cc:290] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:16.799278 22276 consensus_queue.cc:237] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.800124 22143 tablet_bootstrap.cc:654] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.803551 21900 raft_consensus.cc:2463] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:16.804520 21980 leader_election.cc:304] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:16.805274 22211 raft_consensus.cc:2798] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:16.805663 22211 raft_consensus.cc:695] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:16.806334 22211 consensus_queue.cc:237] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.815541 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
W20250112 02:57:16.816758 21978 leader_election.cc:343] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.819536 22145 tablet_bootstrap.cc:654] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.831857 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:16.839926 22143 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.840337 22143 ts_tablet_manager.cc:1397] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.001s
I20250112 02:57:16.841936 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:16.843042 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:16.843211 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:16.844142 22152 tablet_replica.cc:331] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:16.845121 22152 raft_consensus.cc:2238] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:16.845643 22152 raft_consensus.cc:2267] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:16.848148 22152 ts_tablet_manager.cc:1905] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:16.850596 22143 raft_consensus.cc:357] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.851297 22143 raft_consensus.cc:383] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.851552 22143 raft_consensus.cc:738] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.852121 22143 consensus_queue.cc:260] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.861155 22153 ts_tablet_manager.cc:1918] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:16.861513 22153 log.cc:1198] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:16.862104 22153 ts_tablet_manager.cc:1939] T 1f77f8b613544e909ed3397c4b2ebd08 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:16.868256 22143 ts_tablet_manager.cc:1428] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20250112 02:57:16.872313 22143 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.894969 22143 tablet_bootstrap.cc:654] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.896075 22151 ts_tablet_manager.cc:1918] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:16.896400 22151 log.cc:1198] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/b1e608afe71349f1ae609e584022e510
I20250112 02:57:16.907610 22151 ts_tablet_manager.cc:1939] T b1e608afe71349f1ae609e584022e510 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:16.911234 22144 tablet_bootstrap.cc:492] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:16.911753 22144 ts_tablet_manager.cc:1397] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.371s user 0.066s sys 0.149s
I20250112 02:57:16.911983 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet b1e608afe71349f1ae609e584022e510 (table default.Table_3827b9f8c5ad4db7865b690a90b4e7d4 [id=32f1cf819ea748d78535c9c44d64b92e]) successfully deleted
W20250112 02:57:16.912818 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5890ebb3c55943d5a334d6c93fbdaf0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5890ebb3c55943d5a334d6c93fbdaf0e already in progress: creating tablet
I20250112 02:57:16.914168 22144 raft_consensus.cc:357] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.914891 22144 raft_consensus.cc:383] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.915169 22144 raft_consensus.cc:738] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.915874 22144 consensus_queue.cc:260] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:16.918903 21388 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6fbe0e20a157479e9160fc6782e3bf37 has no consensus state
W20250112 02:57:16.919503 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77d80c6a60bc46449c7c0656128100c8 has no consensus state
I20250112 02:57:16.920164 21391 master_service.cc:946] ReplaceTablet: received request to replace tablet a6bfe4c753f84aeeba7446def9898d43 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:16.920194 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ca45fbb115164e9696823db6a8dfda60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:16.922328 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca45fbb115164e9696823db6a8dfda60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:16.923635 22144 ts_tablet_manager.cc:1428] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250112 02:57:16.930246 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 1f77f8b613544e909ed3397c4b2ebd08 (table default.Table_355d103eff5a46c2bd4aedc3e85aa88a [id=1322b48e2bf14cf2851b7f0e1828441c]) successfully deleted
I20250112 02:57:16.932291 22143 tablet_bootstrap.cc:492] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:16.932811 22143 ts_tablet_manager.cc:1397] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.001s
I20250112 02:57:16.933342 22144 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:16.938403 22144 tablet_bootstrap.cc:654] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:16.939860 22152 ts_tablet_manager.cc:1918] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:16.940196 22152 log.cc:1198] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:16.940784 22152 ts_tablet_manager.cc:1939] T 5890ebb3c55943d5a334d6c93fbdaf0e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:16.944715 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 402c24e54ab040e29949b1e4efbe5ff3 from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:16.950809 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5890ebb3c55943d5a334d6c93fbdaf0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5890ebb3c55943d5a334d6c93fbdaf0e already in progress: creating tablet
I20250112 02:57:16.952962 22143 raft_consensus.cc:357] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.953787 22143 raft_consensus.cc:383] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:16.954080 22143 raft_consensus.cc:738] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:16.954942 22143 consensus_queue.cc:260] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.958191 22143 ts_tablet_manager.cc:1428] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.025s user 0.000s sys 0.005s
I20250112 02:57:16.959817 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 5890ebb3c55943d5a334d6c93fbdaf0e (table default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]) successfully deleted
I20250112 02:57:16.973883 22211 raft_consensus.cc:491] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.974493 22211 raft_consensus.cc:513] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.979530 22211 leader_election.cc:290] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:16.978713 22143 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:16.981148 22161 raft_consensus.cc:491] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.981542 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.981545 22161 raft_consensus.cc:513] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.982743 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:16.983287 22161 leader_election.cc:290] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:16.984674 21980 leader_election.cc:343] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.984804 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:16.985581 22044 raft_consensus.cc:2388] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
W20250112 02:57:16.987354 21978 leader_election.cc:343] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:16.987269 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:16.987761 21978 leader_election.cc:304] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:16.988792 21691 leader_election.cc:304] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:16.989521 22161 raft_consensus.cc:3054] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:16.993492 22211 raft_consensus.cc:2743] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.995729 22145 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:16.995931 22161 raft_consensus.cc:2743] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:16.997596 22273 raft_consensus.cc:491] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:16.998036 22145 ts_tablet_manager.cc:1397] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.209s user 0.008s sys 0.003s
I20250112 02:57:16.997998 22273 raft_consensus.cc:513] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:16.999886 22273 leader_election.cc:290] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:17.000872 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:17.001166 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb324cd3120e45449e0500c99338135d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:17.001866 21900 raft_consensus.cc:2388] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:17.002477 22145 raft_consensus.cc:357] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.003165 22145 raft_consensus.cc:383] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.002959 21691 leader_election.cc:304] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:17.003393 22145 raft_consensus.cc:738] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.004635 22273 raft_consensus.cc:3054] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.005443 22143 tablet_bootstrap.cc:654] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.006755 22145 consensus_queue.cc:260] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.010929 22273 raft_consensus.cc:2743] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:17.021117 22143 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:17.021507 22143 ts_tablet_manager.cc:1397] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.001s
I20250112 02:57:17.035404 22276 consensus_queue.cc:1035] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.035404 22211 consensus_queue.cc:1035] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:57:17.046654 22143 raft_consensus.cc:357] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.047318 22143 raft_consensus.cc:383] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.047574 22143 raft_consensus.cc:738] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.048051 22145 ts_tablet_manager.cc:1428] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.050s user 0.004s sys 0.000s
I20250112 02:57:17.051278 22143 consensus_queue.cc:260] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.055989 22143 ts_tablet_manager.cc:1428] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250112 02:57:17.055938 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd (DEFAULT_TABLE table=default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.056838 22143 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:17.057042 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d2be215ded4b8a931339af5ef9b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.063563 22257 consensus_queue.cc:1035] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.066275 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2ac91c5554a64936f364e4e1978f8 (DEFAULT_TABLE table=default.Table_c18efd91f8804ad69b7532dbba118163 [id=17db4776dff34dda81c10c9fe47e9fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.067358 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d2ac91c5554a64936f364e4e1978f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.068430 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd (DEFAULT_TABLE table=default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.069449 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d2be215ded4b8a931339af5ef9b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.079102 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd (DEFAULT_TABLE table=default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.079615 22250 raft_consensus.cc:491] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.080132 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64d2be215ded4b8a931339af5ef9b8dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.080016 22250 raft_consensus.cc:513] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.082930 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748]). Expected version 2 got 0
I20250112 02:57:17.083676 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:17.084033 22250 leader_election.cc:290] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:17.084336 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:17.085580 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d699356dd3341338670b433a7fa02ad" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:17.085908 22250 consensus_queue.cc:1035] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.086210 21756 raft_consensus.cc:2463] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:17.087168 21978 leader_election.cc:304] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:17.087872 22276 raft_consensus.cc:2798] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:17.088177 22276 raft_consensus.cc:491] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:17.088474 22276 raft_consensus.cc:3054] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.088882 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2ac91c5554a64936f364e4e1978f8 (DEFAULT_TABLE table=default.Table_c18efd91f8804ad69b7532dbba118163 [id=17db4776dff34dda81c10c9fe47e9fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.089901 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d2ac91c5554a64936f364e4e1978f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.090231 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a (DEFAULT_TABLE table=default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.091249 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b39d0cf44f4e91908b8e568df0a16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.094022 22144 tablet_bootstrap.cc:492] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:17.094403 22144 ts_tablet_manager.cc:1397] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.170s user 0.019s sys 0.031s
I20250112 02:57:17.095261 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a (DEFAULT_TABLE table=default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.096215 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b39d0cf44f4e91908b8e568df0a16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.096861 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 82d2ac91c5554a64936f364e4e1978f8 (DEFAULT_TABLE table=default.Table_c18efd91f8804ad69b7532dbba118163 [id=17db4776dff34dda81c10c9fe47e9fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.097951 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82d2ac91c5554a64936f364e4e1978f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.099149 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a (DEFAULT_TABLE table=default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:17.100049 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35b39d0cf44f4e91908b8e568df0a16a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:17.102584 22145 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:17.106015 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a15b58bd8c49483082451ae033d064b7" } modify_external_catalogs: true
I20250112 02:57:17.106662 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a15b58bd8c49483082451ae033d064b7" } modify_external_catalogs: true
I20250112 02:57:17.107090 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d699356dd3341338670b433a7fa02ad" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:17.107841 22143 tablet_bootstrap.cc:654] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.114362 21384 catalog_manager.cc:5526] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
W20250112 02:57:17.118868 21980 leader_election.cc:343] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.123397 21390 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_c41d4357625c47ccb556d7a2231c65fd" } new_table_name: "default.Table_8e38f61a93f745f28bf8f5513ba8e4ff" modify_external_catalogs: true
I20250112 02:57:17.124009 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_5ffe090e2d664579985eb72894c11d10" } new_table_name: "default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c" modify_external_catalogs: true
I20250112 02:57:17.117743 21384 catalog_manager.cc:5526] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:17.127622 21384 catalog_manager.cc:5526] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:17.146674 22276 raft_consensus.cc:513] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.153029 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d699356dd3341338670b433a7fa02ad" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:17.153784 21755 raft_consensus.cc:3054] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.130965 21384 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:17.158584 21384 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:17.159273 21384 catalog_manager.cc:5526] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:17.160909 21755 raft_consensus.cc:2463] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:17.161180 21384 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:17.162074 21978 leader_election.cc:304] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:17.162892 22257 raft_consensus.cc:2798] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:17.163331 22257 raft_consensus.cc:695] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:17.164028 22257 consensus_queue.cc:237] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.171473 22276 leader_election.cc:290] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:17.177279 22144 raft_consensus.cc:357] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.177984 22144 raft_consensus.cc:383] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.178213 22144 raft_consensus.cc:738] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.180153 21756 raft_consensus.cc:3054] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.186657 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d699356dd3341338670b433a7fa02ad" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:17.187295 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_c8cd0705965747f2be723fcc60928932"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.188902 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8cd0705965747f2be723fcc60928932 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.191336 22144 consensus_queue.cc:260] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.192781 21900 raft_consensus.cc:3054] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.204741 22145 tablet_bootstrap.cc:654] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.211620 21900 raft_consensus.cc:2463] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:17.212193 22257 consensus_queue.cc:1035] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.214792 22259 raft_consensus.cc:491] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.215216 22259 raft_consensus.cc:513] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.216948 22259 leader_election.cc:290] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:17.218911 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:17.223060 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7de9ed3d1552490da7ab76037d7debc0" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:17.223917 21755 raft_consensus.cc:2388] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:17.224874 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:17.224864 21835 leader_election.cc:304] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:17.225751 22259 raft_consensus.cc:3054] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.226291 21387 catalog_manager.cc:6596] ReplaceTablet: tablet 402c24e54ab040e29949b1e4efbe5ff3 deleted and replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:17.228097 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:17.228803 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:17.230579 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:17.232148 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 402c24e54ab040e29949b1e4efbe5ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402c24e54ab040e29949b1e4efbe5ff3 already in progress: creating tablet
I20250112 02:57:17.232690 22259 raft_consensus.cc:2743] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:17.234091 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 402c24e54ab040e29949b1e4efbe5ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402c24e54ab040e29949b1e4efbe5ff3 already in progress: creating tablet
W20250112 02:57:17.234800 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 402c24e54ab040e29949b1e4efbe5ff3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 402c24e54ab040e29949b1e4efbe5ff3 already in progress: creating tablet
I20250112 02:57:17.243624 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_9821bdb412d34479aff0378b50fc8890"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.246089 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9821bdb412d34479aff0378b50fc8890 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.251178 22144 ts_tablet_manager.cc:1428] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.157s user 0.005s sys 0.000s
I20250112 02:57:17.251919 22143 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:17.252344 22143 ts_tablet_manager.cc:1397] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.196s user 0.014s sys 0.003s
I20250112 02:57:17.254427 22144 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:17.273092 21690 consensus_peers.cc:487] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cad224c5ad844512b50da5190cfd1861. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:17.277434 22276 consensus_queue.cc:1035] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.279915 22144 tablet_bootstrap.cc:654] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.292544 21391 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8a906380b92c40559f248cae72f8be4b
I20250112 02:57:17.301733 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e3131ae5453b4ce58463562eeb65edec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:17.303826 22145 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:17.304363 22145 ts_tablet_manager.cc:1397] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.202s user 0.009s sys 0.000s
W20250112 02:57:17.304549 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3131ae5453b4ce58463562eeb65edec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.310640 22144 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:17.311000 22144 ts_tablet_manager.cc:1397] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.057s user 0.015s sys 0.000s
I20250112 02:57:17.313745 22144 raft_consensus.cc:357] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.314388 22144 raft_consensus.cc:383] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.314702 22144 raft_consensus.cc:738] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.317790 22143 raft_consensus.cc:357] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.318477 22145 raft_consensus.cc:357] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.319135 22145 raft_consensus.cc:383] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.319387 22145 raft_consensus.cc:738] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.320039 22144 consensus_queue.cc:260] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:17.329182 21390 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15c16031f5b74cf9885fac110f445e1a consensus state has no leader
I20250112 02:57:17.329202 22145 consensus_queue.cc:260] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.330009 21390 master_service.cc:946] ReplaceTablet: received request to replace tablet bb324cd3120e45449e0500c99338135d from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:17.340301 22145 ts_tablet_manager.cc:1428] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20250112 02:57:17.336686 22273 consensus_queue.cc:1035] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.361168 22259 raft_consensus.cc:491] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.361594 22259 raft_consensus.cc:513] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.363523 22259 leader_election.cc:290] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:17.367777 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:17.368346 22045 raft_consensus.cc:2463] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:17.369609 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:17.370550 22143 raft_consensus.cc:383] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.370829 22143 raft_consensus.cc:738] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.370786 21836 leader_election.cc:304] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:17.371372 22143 consensus_queue.cc:260] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:17.372675 21835 leader_election.cc:343] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.374737 22259 raft_consensus.cc:2798] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:17.375056 22259 raft_consensus.cc:491] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:17.375383 22259 raft_consensus.cc:3054] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.383719 22145 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:17.386770 22143 ts_tablet_manager.cc:1428] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.116s user 0.006s sys 0.000s
I20250112 02:57:17.387605 22143 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:17.390481 22144 ts_tablet_manager.cc:1428] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.079s user 0.006s sys 0.000s
I20250112 02:57:17.398064 22144 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:17.394279 22259 raft_consensus.cc:513] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.408968 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
W20250112 02:57:17.415269 21835 leader_election.cc:343] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.419790 22144 tablet_bootstrap.cc:654] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.425436 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:17.425964 22045 raft_consensus.cc:3054] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.432176 22045 raft_consensus.cc:2463] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:17.450969 22161 consensus_queue.cc:1035] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:57:17.451238 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c4998c034e54b1bae0ef132cef5b04d has no consensus state
I20250112 02:57:17.456979 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.458822 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.463043 22276 consensus_queue.cc:1035] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.496201 22143 tablet_bootstrap.cc:654] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.509406 22145 tablet_bootstrap.cc:654] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.512678 22257 consensus_queue.cc:1035] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.520608 22143 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:17.521010 22143 ts_tablet_manager.cc:1397] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.134s user 0.008s sys 0.010s
I20250112 02:57:17.523712 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ca45fbb115164e9696823db6a8dfda60" } modify_external_catalogs: true
I20250112 02:57:17.524214 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ca45fbb115164e9696823db6a8dfda60" } modify_external_catalogs: true
I20250112 02:57:17.525554 22143 raft_consensus.cc:357] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.526136 22143 raft_consensus.cc:383] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.526374 22143 raft_consensus.cc:738] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.527194 22143 consensus_queue.cc:260] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.433269 21836 leader_election.cc:304] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
W20250112 02:57:17.533154 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 100023us
I20250112 02:57:17.533353 21390 master_service.cc:946] ReplaceTablet: received request to replace tablet 402c24e54ab040e29949b1e4efbe5ff3 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:17.534086 22154 raft_consensus.cc:2798] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:17.537643 21385 catalog_manager.cc:5526] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:17.539283 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:17.539971 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:17.420128 22259 leader_election.cc:290] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:17.540899 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748]). Expected version 2 got 0
I20250112 02:57:17.541743 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:17.541811 21387 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_8f7412e5cea04b119aed9eab7eb1e8f9" } new_table_name: "default.Table_02f0da840c0646d59412dcb267a4d571" modify_external_catalogs: true
I20250112 02:57:17.543756 21386 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 82) took 1212 ms. Trace:
I20250112 02:57:17.547271 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:17.547930 22151 tablet_replica.cc:331] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:17.548122 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:17.548620 22151 raft_consensus.cc:2238] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:17.548766 22152 tablet_replica.cc:331] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:17.549420 22151 raft_consensus.cc:2267] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:17.549558 22152 raft_consensus.cc:2238] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:17.549973 22152 raft_consensus.cc:2267] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:17.550694 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:17.552392 22153 tablet_replica.cc:331] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:17.552353 22152 ts_tablet_manager.cc:1905] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:17.553056 22153 raft_consensus.cc:2238] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:17.553499 22153 raft_consensus.cc:2267] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:17.555799 22153 ts_tablet_manager.cc:1905] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:17.559803 22151 ts_tablet_manager.cc:1905] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:17.563139 22154 raft_consensus.cc:695] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:17.563933 22154 consensus_queue.cc:237] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.544086 21386 rpcz_store.cc:276] 0112 02:57:16.331112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:16.331268 (+ 156us) service_pool.cc:225] Handling call
0112 02:57:16.375171 (+ 43903us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d8616 00007f7eed140b44 00007f7eed140ad2 00007f7eed140a01 00007f7eed14097c 00007f7eed13fc32 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:16.376256 (+ 1085us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:16.377310 (+ 1054us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:16.377459 (+ 149us) catalog_manager.cc:2762] Locking tablets
0112 02:57:16.504416 (+126957us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:16.829601 (+325185us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:16.829822 (+ 221us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:17.541659 (+711837us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:17.543696 (+ 2037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:16.693049 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:16.693512 (+ 463us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:16.693534 (+ 22us) write_op.cc:435] Acquired schema lock
0112 02:57:16.693554 (+ 20us) tablet.cc:659] Decoding operations
0112 02:57:16.693867 (+ 313us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:16.693909 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:16.693930 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:16.694176 (+ 246us) tablet.cc:698] Row locks acquired
0112 02:57:16.694189 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:16.694301 (+ 112us) write_op.cc:270] Start()
0112 02:57:16.694379 (+ 78us) write_op.cc:276] Timestamp: P: 1736650636694319 usec, L: 0
0112 02:57:16.694393 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:16.695026 (+ 633us) log.cc:844] Serialized 733 byte log entry
0112 02:57:16.827718 (+132692us) op_driver.cc:464] REPLICATION: finished
0112 02:57:16.828390 (+ 672us) write_op.cc:301] APPLY: starting
0112 02:57:16.828446 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:16.828546 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:16.828645 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:16.828827 (+ 182us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:16.828978 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:16.829015 (+ 37us) write_op.cc:312] APPLY: finished
0112 02:57:16.829176 (+ 161us) log.cc:844] Serialized 28 byte log entry
0112 02:57:16.829316 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:16.829377 (+ 61us) write_op.cc:454] Released schema lock
0112 02:57:16.829403 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:16.829993 (+ 590us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:16.830014 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":97973120,"child_traces":[["op",{"apply.queue_time_us":501,"apply.run_cpu_time_us":1213,"apply.run_wall_time_us":1580,"num_ops":2,"prepare.queue_time_us":186698,"prepare.run_cpu_time_us":2268,"prepare.run_wall_time_us":47867,"raft.queue_time_us":3472,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":4336,"replication_time_us":133220}]]}
I20250112 02:57:17.570760 22143 ts_tablet_manager.cc:1428] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250112 02:57:17.572732 22257 consensus_queue.cc:1035] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:57:17.574020 22143 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:17.583508 22161 consensus_queue.cc:1035] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:17.598975 22144 tablet_bootstrap.cc:492] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:17.599509 22144 ts_tablet_manager.cc:1397] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.202s user 0.009s sys 0.000s
I20250112 02:57:17.602079 22257 raft_consensus.cc:491] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.603309 22257 raft_consensus.cc:513] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.605674 22152 ts_tablet_manager.cc:1918] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:17.605804 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:17.606475 21756 raft_consensus.cc:2463] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:17.607168 22143 tablet_bootstrap.cc:654] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.609189 22153 ts_tablet_manager.cc:1918] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:17.609623 22153 log.cc:1198] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:17.607307 21978 leader_election.cc:304] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:17.613004 22276 raft_consensus.cc:2798] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:17.613329 22276 raft_consensus.cc:491] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:17.613675 22276 raft_consensus.cc:3054] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.613476 22152 log.cc:1198] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:17.614620 22152 ts_tablet_manager.cc:1939] T a6bfe4c753f84aeeba7446def9898d43 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:17.615092 22151 ts_tablet_manager.cc:1918] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:17.615468 22151 log.cc:1198] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:17.616005 22145 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:17.616477 22145 ts_tablet_manager.cc:1397] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.233s user 0.000s sys 0.011s
I20250112 02:57:17.618031 22151 ts_tablet_manager.cc:1939] T a6bfe4c753f84aeeba7446def9898d43 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:17.620096 22257 leader_election.cc:290] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:17.619843 22276 raft_consensus.cc:513] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.620316 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet a6bfe4c753f84aeeba7446def9898d43 (table default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]) successfully deleted
I20250112 02:57:17.621930 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:17.622072 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet a6bfe4c753f84aeeba7446def9898d43 (table default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]) successfully deleted
I20250112 02:57:17.622485 22276 leader_election.cc:290] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
W20250112 02:57:17.623164 21980 leader_election.cc:343] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.624233 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:17.625298 21980 leader_election.cc:343] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.625172 22144 raft_consensus.cc:357] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.625499 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:17.625821 22144 raft_consensus.cc:383] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.626062 22144 raft_consensus.cc:738] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.626142 21756 raft_consensus.cc:3054] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.626267 22143 tablet_bootstrap.cc:492] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:17.626735 22143 ts_tablet_manager.cc:1397] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.001s
I20250112 02:57:17.628774 22143 raft_consensus.cc:357] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.633282 22143 raft_consensus.cc:383] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.633570 22143 raft_consensus.cc:738] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.634131 22143 consensus_queue.cc:260] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.633277 21756 raft_consensus.cc:2463] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:17.637621 21384 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748] alter complete (version 2)
I20250112 02:57:17.637786 22143 ts_tablet_manager.cc:1428] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250112 02:57:17.627019 22144 consensus_queue.cc:260] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.642940 21978 leader_election.cc:304] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:17.610432 22153 ts_tablet_manager.cc:1939] T a6bfe4c753f84aeeba7446def9898d43 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:17.645169 22143 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:17.646471 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet a6bfe4c753f84aeeba7446def9898d43 (table default.Table_f6351a54c2544bfe94a9b4c219f69ad5 [id=9ae3f979b81c4b29b3e49873e58054a3]) successfully deleted
I20250112 02:57:17.649585 22276 raft_consensus.cc:2798] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:17.650772 22143 tablet_bootstrap.cc:654] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.653968 22144 ts_tablet_manager.cc:1428] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20250112 02:57:17.657630 22276 raft_consensus.cc:695] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:17.658528 22276 consensus_queue.cc:237] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.683171 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_2c8b88dcd37b4ecfa44453ce970e61cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.685557 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c8b88dcd37b4ecfa44453ce970e61cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.686854 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a77c396c9dd148f4aea2dc0bce0ec411" } new_table_name: "default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea" modify_external_catalogs: true
I20250112 02:57:17.704895 22336 consensus_queue.cc:1035] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:57:17.736336 22144 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:17.673520 22145 raft_consensus.cc:357] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.746961 22145 raft_consensus.cc:383] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.747357 22145 raft_consensus.cc:738] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.748167 22145 consensus_queue.cc:260] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.753409 21388 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ae76dbde59d541ecb4b8828b2969075a
I20250112 02:57:17.754660 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_4dc1a81866be4dad82a2dc48e27f273a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.756054 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc1a81866be4dad82a2dc48e27f273a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.758890 22145 ts_tablet_manager.cc:1428] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.142s user 0.005s sys 0.002s
I20250112 02:57:17.721973 22345 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:17.789472 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3c4998c034e54b1bae0ef132cef5b04d has no consensus state
I20250112 02:57:17.790306 22154 consensus_queue.cc:1035] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.790853 22211 consensus_queue.cc:1035] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.767570 22145 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:17.795084 22351 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:17.798231 22144 tablet_bootstrap.cc:654] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.800957 22145 tablet_bootstrap.cc:654] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.798190 22346 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:17.815547 22143 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:17.815918 22143 ts_tablet_manager.cc:1397] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.016s
I20250112 02:57:17.842772 22145 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:17.843679 22145 ts_tablet_manager.cc:1397] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.001s
I20250112 02:57:17.844482 22144 tablet_bootstrap.cc:492] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:17.844862 22144 ts_tablet_manager.cc:1397] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.109s user 0.009s sys 0.001s
I20250112 02:57:17.846189 22145 raft_consensus.cc:357] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.846835 22145 raft_consensus.cc:383] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.847112 22145 raft_consensus.cc:738] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.849042 22145 consensus_queue.cc:260] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.853483 22143 raft_consensus.cc:357] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.854161 22143 raft_consensus.cc:383] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.863690 22211 raft_consensus.cc:491] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.862847 21390 catalog_manager.cc:5526] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:17.864102 22211 raft_consensus.cc:513] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.864595 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_8e38f61a93f745f28bf8f5513ba8e4ff [id=682416c8b9f6464789d238eb4e2a1c96]). Expected version 1 got 0
I20250112 02:57:17.866134 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:17.866897 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:17.867619 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402c24e54ab040e29949b1e4efbe5ff3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:17.868031 21390 catalog_manager.cc:5526] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:17.869400 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
W20250112 02:57:17.827467 22347 log.cc:926] Time spent T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250112 02:57:17.854418 22143 raft_consensus.cc:738] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.875284 22143 consensus_queue.cc:260] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.866400 22211 leader_election.cc:290] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
W20250112 02:57:17.876510 21980 leader_election.cc:343] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.876780 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402c24e54ab040e29949b1e4efbe5ff3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:17.878029 22144 raft_consensus.cc:357] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.881886 21755 raft_consensus.cc:2463] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:17.888804 22145 ts_tablet_manager.cc:1428] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20250112 02:57:17.890044 22144 raft_consensus.cc:383] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:17.890321 22144 raft_consensus.cc:738] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:17.892412 22144 consensus_queue.cc:260] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:17.893664 21379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111) [suppressed 8 similar messages]
I20250112 02:57:17.894505 22144 ts_tablet_manager.cc:1428] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250112 02:57:17.889922 21978 leader_election.cc:304] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:17.903892 22211 raft_consensus.cc:2798] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:17.904246 22211 raft_consensus.cc:491] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:17.904551 22211 raft_consensus.cc:3054] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.906279 22144 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:17.916750 22250 raft_consensus.cc:491] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:17.914255 22211 raft_consensus.cc:513] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.919425 22211 leader_election.cc:290] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:17.922223 22144 tablet_bootstrap.cc:654] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:17.877869 22143 ts_tablet_manager.cc:1428] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.062s user 0.006s sys 0.000s
I20250112 02:57:17.927666 22143 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:17.927670 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_0192ff1911b647d3bb78166de4f2fc72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:17.928999 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0192ff1911b647d3bb78166de4f2fc72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:17.870299 21390 catalog_manager.cc:5526] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:17.932991 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402c24e54ab040e29949b1e4efbe5ff3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:17.933557 21901 raft_consensus.cc:3054] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:17.917228 22250 raft_consensus.cc:513] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:17.942070 22250 leader_election.cc:290] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:17.943847 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d2be215ded4b8a931339af5ef9b8dd" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:17.945968 21980 leader_election.cc:343] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.948423 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d2be215ded4b8a931339af5ef9b8dd" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:17.950258 21978 leader_election.cc:343] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:17.950647 21978 leader_election.cc:304] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:17.952067 22276 raft_consensus.cc:2743] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:17.957190 22211 consensus_queue.cc:1035] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.958698 22276 consensus_queue.cc:1035] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:17.961081 22143 tablet_bootstrap.cc:654] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:17.968066 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 74450.3us
I20250112 02:57:17.987380 22145 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:18.002748 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "402c24e54ab040e29949b1e4efbe5ff3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:18.003602 21757 raft_consensus.cc:3054] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.005417 22143 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.005779 22143 ts_tablet_manager.cc:1397] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.078s user 0.007s sys 0.003s
I20250112 02:57:18.010071 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ae37aca456fc4eb9a0dd51f067f4b34c" } modify_external_catalogs: true
I20250112 02:57:18.010017 21901 raft_consensus.cc:2463] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:18.010669 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ae37aca456fc4eb9a0dd51f067f4b34c" } modify_external_catalogs: true
I20250112 02:57:18.011777 22143 raft_consensus.cc:357] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.012391 22143 raft_consensus.cc:383] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.012630 22143 raft_consensus.cc:738] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.027591 21980 leader_election.cc:304] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:18.027674 22143 consensus_queue.cc:260] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.028280 22250 raft_consensus.cc:2798] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:18.028764 22250 raft_consensus.cc:695] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:18.029562 22250 consensus_queue.cc:237] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.030797 22143 ts_tablet_manager.cc:1428] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250112 02:57:18.031517 22143 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.033036 22273 raft_consensus.cc:491] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.033551 22273 raft_consensus.cc:513] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.035475 22273 leader_election.cc:290] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.036401 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5890ebb3c55943d5a334d6c93fbdaf0e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:18.036996 21900 raft_consensus.cc:2463] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:18.039085 21690 leader_election.cc:343] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:18.039954 21691 leader_election.cc:304] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters: e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:18.040813 22273 raft_consensus.cc:2798] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:18.041098 22273 raft_consensus.cc:491] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:18.041391 22273 raft_consensus.cc:3054] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.042012 22145 tablet_bootstrap.cc:654] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.047403 22273 raft_consensus.cc:513] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:18.052330 21690 leader_election.cc:343] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:18.057890 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5890ebb3c55943d5a334d6c93fbdaf0e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:18.058456 21901 raft_consensus.cc:3054] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.058437 22273 leader_election.cc:290] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.065164 21757 raft_consensus.cc:2463] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:18.068149 22250 raft_consensus.cc:491] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.068547 22250 raft_consensus.cc:513] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.071774 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:18.072809 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:18.073053 21978 leader_election.cc:343] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:18.074229 21980 leader_election.cc:343] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:18.074633 21980 leader_election.cc:304] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:18.075404 22276 raft_consensus.cc:2743] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:18.077276 22143 tablet_bootstrap.cc:654] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.080199 22250 leader_election.cc:290] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:18.081866 22144 tablet_bootstrap.cc:492] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:18.082229 22144 ts_tablet_manager.cc:1397] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.176s user 0.015s sys 0.014s
I20250112 02:57:18.090720 22144 raft_consensus.cc:357] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.091370 22144 raft_consensus.cc:383] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.091607 22144 raft_consensus.cc:738] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.092159 22144 consensus_queue.cc:260] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.099442 22144 ts_tablet_manager.cc:1428] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250112 02:57:18.109541 22144 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:18.115628 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250112 02:57:18.132517 22273 raft_consensus.cc:491] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.132949 22273 raft_consensus.cc:513] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.135181 22273 leader_election.cc:290] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.135473 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:18.136222 21902 raft_consensus.cc:2463] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:18.136464 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:18.137042 22044 raft_consensus.cc:2463] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:18.137171 21691 leader_election.cc:304] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:18.137691 22143 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.138070 22143 ts_tablet_manager.cc:1397] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.107s user 0.011s sys 0.000s
I20250112 02:57:18.138096 22273 raft_consensus.cc:2798] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:18.138453 22273 raft_consensus.cc:491] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:18.138759 22273 raft_consensus.cc:3054] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.140444 22143 raft_consensus.cc:357] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.141113 22143 raft_consensus.cc:383] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.141374 22143 raft_consensus.cc:738] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.142737 22144 tablet_bootstrap.cc:654] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.136575 22145 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:18.144743 22145 ts_tablet_manager.cc:1397] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.158s user 0.012s sys 0.000s
I20250112 02:57:18.144568 22273 raft_consensus.cc:513] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.146931 22145 raft_consensus.cc:357] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.147557 22145 raft_consensus.cc:383] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.147787 22145 raft_consensus.cc:738] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.148687 22273 leader_election.cc:290] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.148357 22145 consensus_queue.cc:260] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.148846 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:18.149459 22044 raft_consensus.cc:3054] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.157059 22044 raft_consensus.cc:2463] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:18.158118 21690 leader_election.cc:304] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:18.141947 22143 consensus_queue.cc:260] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.158880 22273 raft_consensus.cc:2798] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:18.159281 22273 raft_consensus.cc:695] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:18.160480 22143 ts_tablet_manager.cc:1428] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250112 02:57:18.160032 22273 consensus_queue.cc:237] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.161237 22143 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.174399 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3781431324bc479d97bd22b6bb1243b9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:18.174985 21902 raft_consensus.cc:3054] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.171844 22145 ts_tablet_manager.cc:1428] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250112 02:57:18.180977 22276 consensus_queue.cc:1035] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:18.184240 22143 tablet_bootstrap.cc:654] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.196218 22144 tablet_bootstrap.cc:492] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:18.196676 22144 ts_tablet_manager.cc:1397] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.096s user 0.011s sys 0.004s
I20250112 02:57:18.199532 22144 raft_consensus.cc:357] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.200240 22144 raft_consensus.cc:383] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.200518 22144 raft_consensus.cc:738] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.201195 22144 consensus_queue.cc:260] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.210719 22144 ts_tablet_manager.cc:1428] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250112 02:57:18.207271 21902 raft_consensus.cc:2463] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:18.203418 21901 raft_consensus.cc:2463] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:18.214622 21691 leader_election.cc:304] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters: e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:18.215600 22350 raft_consensus.cc:2798] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:18.215994 22350 raft_consensus.cc:695] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:18.216101 22143 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.216472 22143 ts_tablet_manager.cc:1397] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.001s
I20250112 02:57:18.218036 22144 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:18.216658 22350 consensus_queue.cc:237] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.218786 22143 raft_consensus.cc:357] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.219381 22143 raft_consensus.cc:383] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.219610 22143 raft_consensus.cc:738] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.220252 22143 consensus_queue.cc:260] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.220727 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_8e38f61a93f745f28bf8f5513ba8e4ff [id=682416c8b9f6464789d238eb4e2a1c96]). Expected version 1 got 0
I20250112 02:57:18.221634 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:18.222645 21391 catalog_manager.cc:5526] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:18.224783 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:18.225566 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:18.230772 22276 consensus_queue.cc:1035] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:18.231276 22143 ts_tablet_manager.cc:1428] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250112 02:57:18.234846 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:18.237151 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_7c43ca5fe80342cc8bbba92b400d35de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:18.238477 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c43ca5fe80342cc8bbba92b400d35de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:18.239163 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.241592 22144 tablet_bootstrap.cc:654] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.242774 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.244330 22154 raft_consensus.cc:491] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.244769 22154 raft_consensus.cc:513] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.245676 22250 raft_consensus.cc:491] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.246101 22250 raft_consensus.cc:513] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.246800 22154 leader_election.cc:290] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.248206 22250 leader_election.cc:290] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:18.248970 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:18.252512 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:18.254300 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b39d0cf44f4e91908b8e568df0a16a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:18.255636 21980 leader_election.cc:343] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:18.258746 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43c413e1f8fe48039c4dd1b066fdd857" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:18.259449 21756 raft_consensus.cc:2388] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:18.260972 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b39d0cf44f4e91908b8e568df0a16a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:18.261579 21755 raft_consensus.cc:2463] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:18.263118 21835 leader_election.cc:304] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:18.262635 21978 leader_election.cc:304] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:18.264158 22276 raft_consensus.cc:2798] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:18.264401 22154 raft_consensus.cc:3054] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.264485 22276 raft_consensus.cc:491] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:18.264868 22276 raft_consensus.cc:3054] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:18.269809 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 35b39d0cf44f4e91908b8e568df0a16a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35b39d0cf44f4e91908b8e568df0a16a already in progress: creating tablet
I20250112 02:57:18.272096 22144 tablet_bootstrap.cc:492] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:18.272454 22144 ts_tablet_manager.cc:1397] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.000s
I20250112 02:57:18.282212 22154 raft_consensus.cc:2743] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:18.288784 22350 raft_consensus.cc:491] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.289269 22350 raft_consensus.cc:513] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.290388 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 33f89398c7aa4f2a938a476db897c833 (DEFAULT_TABLE table=default.Table_a560e6a044654680a00c3488dc51ce17 [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.291435 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f89398c7aa4f2a938a476db897c833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.292312 21388 master_service.cc:946] ReplaceTablet: received request to replace tablet 35b39d0cf44f4e91908b8e568df0a16a from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:18.300477 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 33f89398c7aa4f2a938a476db897c833 (DEFAULT_TABLE table=default.Table_a560e6a044654680a00c3488dc51ce17 [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.301498 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f89398c7aa4f2a938a476db897c833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.311472 22379 tablet_replica.cc:331] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:18.310212 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:18.312345 22379 raft_consensus.cc:2238] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:18.312793 22379 raft_consensus.cc:2267] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:18.316936 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27234f3e69d544bdb53fb887874c2c0a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:18.317855 22044 raft_consensus.cc:2388] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:18.319118 21690 leader_election.cc:304] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:18.322053 22383 raft_consensus.cc:3054] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.323282 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.324469 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7d4e98b6fa4aecb5955e8603eac779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.324630 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.226595 21391 catalog_manager.cc:5526] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:18.326958 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.327754 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7d4e98b6fa4aecb5955e8603eac779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.327888 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbe0e20a157479e9160fc6782e3bf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.346408 22145 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:18.346956 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 33f89398c7aa4f2a938a476db897c833 (DEFAULT_TABLE table=default.Table_a560e6a044654680a00c3488dc51ce17 [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.348012 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f89398c7aa4f2a938a476db897c833. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.350382 22276 raft_consensus.cc:513] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.353281 22276 leader_election.cc:290] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:18.353945 21388 master_service.cc:946] ReplaceTablet: received request to replace tablet bb324cd3120e45449e0500c99338135d from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:18.355167 22376 tablet_replica.cc:331] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:18.355530 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.356470 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7d4e98b6fa4aecb5955e8603eac779. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.355618 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b39d0cf44f4e91908b8e568df0a16a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
W20250112 02:57:18.358062 21978 leader_election.cc:343] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:18.358407 22376 raft_consensus.cc:2238] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:18.356009 22350 leader_election.cc:290] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.354728 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b39d0cf44f4e91908b8e568df0a16a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:18.367651 21386 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_8f7412e5cea04b119aed9eab7eb1e8f9 [id=ed996933e39d472992d6f351ae51fb39]). Expected version 1 got 0
I20250112 02:57:18.368705 21386 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:18.369865 21386 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:18.371222 21386 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_8e38f61a93f745f28bf8f5513ba8e4ff [id=682416c8b9f6464789d238eb4e2a1c96]). Expected version 1 got 0
I20250112 02:57:18.373351 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_1ab713178ff6472aa38bb55cd1c55d5e" } modify_external_catalogs: true
I20250112 02:57:18.374248 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_1ab713178ff6472aa38bb55cd1c55d5e" } modify_external_catalogs: true
I20250112 02:57:18.274462 22144 raft_consensus.cc:357] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.383297 22144 raft_consensus.cc:383] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.383550 22144 raft_consensus.cc:738] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.384280 22144 consensus_queue.cc:260] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.397615 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.398196 22144 ts_tablet_manager.cc:1428] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.126s user 0.005s sys 0.001s
I20250112 02:57:18.399770 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.402089 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:18.403034 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e608afe71349f1ae609e584022e510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:18.403837 22143 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.410617 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.411341 22375 tablet_replica.cc:331] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:18.412104 22375 raft_consensus.cc:2238] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:18.412559 22375 raft_consensus.cc:2267] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:18.413067 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:18.415550 22375 ts_tablet_manager.cc:1905] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:18.419920 22346 tablet.cc:1721] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:18.376232 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:18.425074 22143 tablet_bootstrap.cc:654] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:18.423753 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:18.379266 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_91b38432112c4a8392b137502ce37881"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:18.429988 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91b38432112c4a8392b137502ce37881 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:18.434286 22379 ts_tablet_manager.cc:1905] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:18.386158 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 27) took 1865 ms (client timeout 1000 ms). Trace:
W20250112 02:57:18.399289 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet a6bfe4c753f84aeeba7446def9898d43 because the tablet was not found. No further retry: Not found: Tablet not found: a6bfe4c753f84aeeba7446def9898d43
W20250112 02:57:18.438585 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 1f77f8b613544e909ed3397c4b2ebd08 because the tablet was not found. No further retry: Not found: Tablet not found: 1f77f8b613544e909ed3397c4b2ebd08
W20250112 02:57:18.443241 21389 rpcz_store.cc:269] 0112 02:57:16.521004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:16.521806 (+ 802us) service_pool.cc:225] Handling call
0112 02:57:18.386065 (+1864259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3086,"reported_tablets":10}
I20250112 02:57:18.440145 22144 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:18.445964 21389 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7f26edfb6fc44a20b7e131ccbabf6f33" } modify_external_catalogs: true
I20250112 02:57:18.450069 22154 consensus_queue.cc:1035] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:18.456735 21389 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7f26edfb6fc44a20b7e131ccbabf6f33" } modify_external_catalogs: true
W20250112 02:57:18.407156 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 20) took 2761 ms (client timeout 1000 ms). Trace:
I20250112 02:57:18.461714 22383 raft_consensus.cc:2743] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:18.460683 21383 rpcz_store.cc:269] 0112 02:57:15.645995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:15.646173 (+ 178us) service_pool.cc:225] Handling call
0112 02:57:18.407081 (+2760908us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:16.371589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:16.372180 (+ 591us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:16.372216 (+ 36us) write_op.cc:435] Acquired schema lock
0112 02:57:16.372237 (+ 21us) tablet.cc:659] Decoding operations
0112 02:57:16.372591 (+ 354us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:16.372642 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:16.372676 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:16.372875 (+ 199us) tablet.cc:698] Row locks acquired
0112 02:57:16.372895 (+ 20us) write_op.cc:260] PREPARE: finished
0112 02:57:16.373046 (+ 151us) write_op.cc:270] Start()
0112 02:57:16.373150 (+ 104us) write_op.cc:276] Timestamp: P: 1736650636373135 usec, L: 0
0112 02:57:16.373171 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 02:57:16.466273 (+ 93102us) log.cc:844] Serialized 535 byte log entry
0112 02:57:16.471467 (+ 5194us) spinlock_profiling.cc:243] Waited 92.7 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b66d1 00007f7eed62d198 00007f7eeda993b1 00007f7eeda98b88 00007f7eeda9d821 00007f7eeda9d7d9 00007f7eeda9d769 00007f7eeda9d731 00007f7eeda9c95d 000000000059ea04 000000000059e839
0112 02:57:16.483875 (+ 12408us) op_driver.cc:464] REPLICATION: finished
0112 02:57:16.484158 (+ 283us) write_op.cc:301] APPLY: starting
0112 02:57:16.484217 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:16.484310 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:16.484331 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:16.484474 (+ 143us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:16.484604 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:16.484663 (+ 59us) write_op.cc:312] APPLY: finished
0112 02:57:16.484836 (+ 173us) log.cc:844] Serialized 22 byte log entry
0112 02:57:16.485542 (+ 706us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:16.485607 (+ 65us) write_op.cc:454] Released schema lock
0112 02:57:16.485637 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:16.597837 (+112200us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:16.598118 (+ 281us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4434,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":146,"apply.run_cpu_time_us":1816,"apply.run_wall_time_us":112421,"num_ops":1,"prepare.queue_time_us":62203,"prepare.run_cpu_time_us":3962,"prepare.run_wall_time_us":100070,"raft.queue_time_us":167,"raft.run_cpu_time_us":662,"raft.run_wall_time_us":11549,"replication_time_us":110612,"spinlock_wait_cycles":213125888}]]}
I20250112 02:57:18.417706 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_2ae6922ba3dd4bbabb88dd71998911a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:18.470981 22144 tablet_bootstrap.cc:654] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:18.418753 21385 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 31) took 1704 ms (client timeout 1000 ms). Trace:
W20250112 02:57:18.472460 21385 rpcz_store.cc:269] 0112 02:57:16.714697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:16.839337 (+124640us) service_pool.cc:225] Handling call
0112 02:57:16.852961 (+ 13624us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:18.418687 (+1565726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:17.809399 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:17.809811 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:17.809838 (+ 27us) write_op.cc:435] Acquired schema lock
0112 02:57:17.809855 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:17.810120 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:17.810162 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:17.810188 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:17.810351 (+ 163us) tablet.cc:698] Row locks acquired
0112 02:57:17.810365 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:17.810481 (+ 116us) write_op.cc:270] Start()
0112 02:57:17.810593 (+ 112us) write_op.cc:276] Timestamp: P: 1736650637810525 usec, L: 0
0112 02:57:17.810610 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:17.820433 (+ 9823us) log.cc:844] Serialized 535 byte log entry
0112 02:57:17.853713 (+ 33280us) op_driver.cc:464] REPLICATION: finished
0112 02:57:17.855451 (+ 1738us) write_op.cc:301] APPLY: starting
0112 02:57:17.855497 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:17.855608 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:17.855706 (+ 98us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:17.855856 (+ 150us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:17.855992 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:17.856061 (+ 69us) write_op.cc:312] APPLY: finished
0112 02:57:17.856220 (+ 159us) log.cc:844] Serialized 22 byte log entry
0112 02:57:17.856355 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.856410 (+ 55us) write_op.cc:454] Released schema lock
0112 02:57:17.856442 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:17.856669 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.856696 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":23989504,"child_traces":[["op",{"apply.queue_time_us":1628,"apply.run_cpu_time_us":1177,"apply.run_wall_time_us":1176,"num_ops":1,"prepare.queue_time_us":240268,"prepare.run_cpu_time_us":2079,"prepare.run_wall_time_us":12426,"replication_time_us":43030}]]}
W20250112 02:57:18.419057 21384 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 21) took 1708 ms (client timeout 1000 ms). Trace:
W20250112 02:57:18.482579 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ae6922ba3dd4bbabb88dd71998911a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:18.489531 22375 ts_tablet_manager.cc:1918] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:18.493938 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.494943 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbe0e20a157479e9160fc6782e3bf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.365520 22145 tablet_bootstrap.cc:654] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.497881 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (DEFAULT_TABLE table=default.Table_8972a68b9140407885f2933d6362b3b3 [id=e6249c87d6424956aab5e1967874bce5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:18.367780 21980 leader_election.cc:343] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:18.499495 21980 leader_election.cc:304] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:18.500285 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 132604us
I20250112 02:57:18.503647 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bca1c68aae24a1f9974a0bd2244fbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.510401 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.510838 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.511566 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e608afe71349f1ae609e584022e510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.514348 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.516773 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.517071 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (DEFAULT_TABLE table=default.Table_8972a68b9140407885f2933d6362b3b3 [id=e6249c87d6424956aab5e1967874bce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.517998 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f77f8b613544e909ed3397c4b2ebd08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:14 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.481062 22385 tablet.cc:1721] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:18.495180 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (DEFAULT_TABLE table=default.Table_8972a68b9140407885f2933d6362b3b3 [id=e6249c87d6424956aab5e1967874bce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.510713 22389 consensus_queue.cc:1035] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:18.522457 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbe0e20a157479e9160fc6782e3bf37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.531908 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bca1c68aae24a1f9974a0bd2244fbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.532833 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bca1c68aae24a1f9974a0bd2244fbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.531762 22375 log.cc:1198] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/cad224c5ad844512b50da5190cfd1861
I20250112 02:57:18.539299 22375 ts_tablet_manager.cc:1939] T cad224c5ad844512b50da5190cfd1861 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:18.568629 22346 tablet.cc:1721] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:18.569276 22399 tablet.cc:1721] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:18.579954 21383 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_8972a68b9140407885f2933d6362b3b3" } new_table_name: "default.Table_7b171c9ac8164714bfeeb9226098e7ff" modify_external_catalogs: true
I20250112 02:57:18.595850 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 64d2be215ded4b8a931339af5ef9b8dd from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:18.358896 22376 raft_consensus.cc:2267] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:18.600309 22211 raft_consensus.cc:2701] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Received election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20250112 02:57:18.611066 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_2974589a1bbd43d0b58eac1aeed1c19f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:18.613173 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2974589a1bbd43d0b58eac1aeed1c19f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:18.618611 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 117985us
I20250112 02:57:18.619273 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet cad224c5ad844512b50da5190cfd1861 (table default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]) successfully deleted
I20250112 02:57:18.620957 22144 tablet_bootstrap.cc:492] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:18.621316 22144 ts_tablet_manager.cc:1397] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.181s user 0.014s sys 0.005s
I20250112 02:57:18.626657 22379 ts_tablet_manager.cc:1918] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:18.627996 22379 log.cc:1198] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:18.627779 22144 raft_consensus.cc:357] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.628381 22144 raft_consensus.cc:383] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.628614 22144 raft_consensus.cc:738] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.629366 22379 ts_tablet_manager.cc:1939] T 35b39d0cf44f4e91908b8e568df0a16a P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:18.629118 22144 consensus_queue.cc:260] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.640524 22379 tablet_replica.cc:331] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:18.642985 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 35b39d0cf44f4e91908b8e568df0a16a (table default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]) successfully deleted
I20250112 02:57:18.643468 22379 raft_consensus.cc:2238] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:18.644411 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet a6bfe4c753f84aeeba7446def9898d43 because the tablet was not found. No further retry: Not found: Tablet not found: a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:18.644588 22144 ts_tablet_manager.cc:1428] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250112 02:57:18.645453 22144 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:18.646396 22379 raft_consensus.cc:2267] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:18.647701 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet b1e608afe71349f1ae609e584022e510 because the tablet was not found. No further retry: Not found: Tablet not found: b1e608afe71349f1ae609e584022e510
I20250112 02:57:18.651926 22390 raft_consensus.cc:491] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.652412 22390 raft_consensus.cc:513] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.653703 22379 ts_tablet_manager.cc:1905] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:18.654021 21389 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 64d2be215ded4b8a931339af5ef9b8dd
I20250112 02:57:18.654254 22390 leader_election.cc:290] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.656919 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:18.657491 21757 raft_consensus.cc:2463] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:18.658368 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:18.659111 22043 raft_consensus.cc:2463] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:18.659873 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:18.663463 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:18.658249 22144 tablet_bootstrap.cc:654] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.687536 22211 consensus_queue.cc:1035] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:57:18.476565 21384 rpcz_store.cc:269] 0112 02:57:16.702278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:16.702421 (+ 143us) service_pool.cc:225] Handling call
0112 02:57:16.735348 (+ 32927us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:18.410441 (+1675093us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:17.463140 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:17.463514 (+ 374us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:17.468695 (+ 5181us) write_op.cc:435] Acquired schema lock
0112 02:57:17.468716 (+ 21us) tablet.cc:659] Decoding operations
0112 02:57:17.471898 (+ 3182us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:17.471940 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:17.471964 (+ 24us) tablet.cc:682] Acquiring locks for 4 operations
0112 02:57:17.472249 (+ 285us) tablet.cc:698] Row locks acquired
0112 02:57:17.472263 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:17.472366 (+ 103us) write_op.cc:270] Start()
0112 02:57:17.472444 (+ 78us) write_op.cc:276] Timestamp: P: 1736650637472486 usec, L: 0
0112 02:57:17.472458 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:17.472922 (+ 464us) log.cc:844] Serialized 1685 byte log entry
0112 02:57:17.526859 (+ 53937us) op_driver.cc:464] REPLICATION: finished
0112 02:57:17.529381 (+ 2522us) write_op.cc:301] APPLY: starting
0112 02:57:17.529446 (+ 65us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:17.529581 (+ 135us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:17.529672 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:17.530012 (+ 340us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:17.530150 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0112 02:57:17.530191 (+ 41us) write_op.cc:312] APPLY: finished
0112 02:57:17.530379 (+ 188us) log.cc:844] Serialized 40 byte log entry
0112 02:57:17.530550 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.530630 (+ 80us) write_op.cc:454] Released schema lock
0112 02:57:17.530662 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:17.530868 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.530897 (+ 29us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:17.610307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:17.610806 (+ 499us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:17.610828 (+ 22us) write_op.cc:435] Acquired schema lock
0112 02:57:17.610840 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:17.611163 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:17.611202 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:17.611222 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:17.612652 (+ 1430us) tablet.cc:698] Row locks acquired
0112 02:57:17.612668 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:17.612772 (+ 104us) write_op.cc:270] Start()
0112 02:57:17.612851 (+ 79us) write_op.cc:276] Timestamp: P: 1736650637612741 usec, L: 0
0112 02:57:17.623170 (+ 10319us) op_driver.cc:348] REPLICATION: starting
0112 02:57:17.627780 (+ 4610us) log.cc:844] Serialized 355 byte log entry
0112 02:57:17.635827 (+ 8047us) op_driver.cc:464] REPLICATION: finished
0112 02:57:17.636170 (+ 343us) write_op.cc:301] APPLY: starting
0112 02:57:17.636220 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:17.636314 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:17.636413 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:17.636555 (+ 142us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:17.636697 (+ 142us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:17.636737 (+ 40us) write_op.cc:312] APPLY: finished
0112 02:57:17.636903 (+ 166us) log.cc:844] Serialized 22 byte log entry
0112 02:57:17.637062 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.637142 (+ 80us) write_op.cc:454] Released schema lock
0112 02:57:17.637172 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:17.695829 (+ 58657us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:17.695880 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":14,"spinlock_wait_cycles":68516736,"child_traces":[["op",{"apply.queue_time_us":2373,"apply.run_cpu_time_us":1457,"apply.run_wall_time_us":1455,"num_ops":4,"prepare.queue_time_us":255248,"prepare.run_cpu_time_us":2188,"prepare.run_wall_time_us":10059,"raft.queue_time_us":2997,"raft.run_cpu_time_us":713,"raft.run_wall_time_us":714,"replication_time_us":54300}],["op",{"apply.queue_time_us":204,"apply.run_cpu_time_us":1177,"apply.run_wall_time_us":1176,"num_ops":1,"prepare.queue_time_us":76776,"prepare.run_cpu_time_us":2082,"prepare.run_wall_time_us":85489,"raft.queue_time_us":80,"raft.run_cpu_time_us":583,"raft.run_wall_time_us":583,"replication_time_us":12566}]]}
I20250112 02:57:18.692660 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_432c0b6f219d4c11b784bd6913108e00" } new_table_name: "default.Table_0d24be46f0ad49dcb9757b12d906a254" modify_external_catalogs: true
I20250112 02:57:18.695386 22376 ts_tablet_manager.cc:1905] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:18.696147 22144 tablet_bootstrap.cc:492] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:18.696493 22144 ts_tablet_manager.cc:1397] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.000s
I20250112 02:57:18.702697 22144 raft_consensus.cc:357] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.703388 22144 raft_consensus.cc:383] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.703646 22144 raft_consensus.cc:738] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.704192 22144 consensus_queue.cc:260] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.726444 22379 ts_tablet_manager.cc:1918] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:18.726958 22379 log.cc:1198] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/cad224c5ad844512b50da5190cfd1861
I20250112 02:57:18.727771 22379 ts_tablet_manager.cc:1939] T cad224c5ad844512b50da5190cfd1861 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:18.730435 22379 tablet_replica.cc:331] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:18.731264 22379 raft_consensus.cc:2238] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:18.731278 21389 master_service.cc:946] ReplaceTablet: received request to replace tablet 27234f3e69d544bdb53fb887874c2c0a from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:18.731772 22379 raft_consensus.cc:2267] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:18.733403 22143 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.733775 22143 ts_tablet_manager.cc:1397] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.330s user 0.018s sys 0.024s
I20250112 02:57:18.734793 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet cad224c5ad844512b50da5190cfd1861 (table default.Table_d5caf55266494547a656e5afaf12bf4b [id=ba613b7c77394874a8e98f875d950c07]) successfully deleted
I20250112 02:57:18.735865 22143 raft_consensus.cc:357] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.736471 22143 raft_consensus.cc:383] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.736733 22143 raft_consensus.cc:738] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.737455 22143 consensus_queue.cc:260] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.749032 22145 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:18.749106 22143 ts_tablet_manager.cc:1428] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250112 02:57:18.749439 22145 ts_tablet_manager.cc:1397] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.403s user 0.013s sys 0.003s
I20250112 02:57:18.750418 22143 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.752293 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 (DEFAULT_TABLE table=default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.752455 22145 raft_consensus.cc:357] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.753149 22145 raft_consensus.cc:383] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.753232 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847b7af9e0b4aaba1f7cd06534f6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.753398 22145 raft_consensus.cc:738] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.764083 22379 ts_tablet_manager.cc:1905] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:18.764577 22145 consensus_queue.cc:260] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.770809 22143 tablet_bootstrap.cc:654] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:18.780124 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 28) took 1079 ms (client timeout 1000 ms). Trace:
I20250112 02:57:18.780771 22145 ts_tablet_manager.cc:1428] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250112 02:57:18.784197 22145 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:18.786036 22143 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.786406 22143 ts_tablet_manager.cc:1397] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.002s
I20250112 02:57:18.788681 22143 raft_consensus.cc:357] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.789357 22143 raft_consensus.cc:383] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.789601 22143 raft_consensus.cc:738] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.790282 22143 consensus_queue.cc:260] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:18.786320 21391 rpcz_store.cc:269] 0112 02:57:17.700777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:17.700910 (+ 133us) service_pool.cc:225] Handling call
0112 02:57:17.712648 (+ 11738us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:18.357003 (+644355us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.357031 (+ 28us) write_op.cc:454] Released schema lock
0112 02:57:18.780053 (+423022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:18.328544 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:18.328871 (+ 327us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:18.328892 (+ 21us) write_op.cc:435] Acquired schema lock
0112 02:57:18.328905 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:18.329188 (+ 283us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:18.329225 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:18.329244 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:18.329466 (+ 222us) tablet.cc:698] Row locks acquired
0112 02:57:18.329479 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:18.329597 (+ 118us) write_op.cc:270] Start()
0112 02:57:18.329676 (+ 79us) write_op.cc:276] Timestamp: P: 1736650638329665 usec, L: 0
0112 02:57:18.329691 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:18.330159 (+ 468us) log.cc:844] Serialized 919 byte log entry
0112 02:57:18.337527 (+ 7368us) op_driver.cc:464] REPLICATION: finished
0112 02:57:18.337922 (+ 395us) write_op.cc:301] APPLY: starting
0112 02:57:18.337983 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:18.338083 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:18.338163 (+ 80us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:18.338356 (+ 193us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:18.338482 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:18.338539 (+ 57us) write_op.cc:312] APPLY: finished
0112 02:57:18.338701 (+ 162us) log.cc:844] Serialized 29 byte log entry
0112 02:57:18.338838 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.338903 (+ 65us) write_op.cc:454] Released schema lock
0112 02:57:18.338930 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":67152,"reported_tablets":13,"spinlock_wait_cycles":19340032,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":1182,"apply.run_wall_time_us":1196,"num_ops":2,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":1984,"prepare.run_wall_time_us":19913,"raft.queue_time_us":143,"raft.run_cpu_time_us":658,"raft.run_wall_time_us":658,"replication_time_us":7753}]]}
I20250112 02:57:18.802363 22143 ts_tablet_manager.cc:1428] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.016s user 0.004s sys 0.000s
I20250112 02:57:18.660909 21835 leader_election.cc:304] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
W20250112 02:57:18.803676 21835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 142785us
I20250112 02:57:18.811220 22379 ts_tablet_manager.cc:1918] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:57:18.816809 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 154942us
I20250112 02:57:18.819260 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.820051 22379 log.cc:1198] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/64d2be215ded4b8a931339af5ef9b8dd
I20250112 02:57:18.819638 22143 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.821830 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 (DEFAULT_TABLE table=default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.823626 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847b7af9e0b4aaba1f7cd06534f6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.826229 22143 tablet_bootstrap.cc:654] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.828378 22145 tablet_bootstrap.cc:654] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.820686 22379 ts_tablet_manager.cc:1939] T 64d2be215ded4b8a931339af5ef9b8dd P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:18.831656 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 64d2be215ded4b8a931339af5ef9b8dd (table default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]) successfully deleted
I20250112 02:57:18.832119 22250 consensus_queue.cc:1035] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:18.834233 22143 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:18.834646 22143 ts_tablet_manager.cc:1397] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20250112 02:57:18.836820 22143 raft_consensus.cc:357] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.837456 22143 raft_consensus.cc:383] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.837698 22143 raft_consensus.cc:738] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:18.838397 22143 consensus_queue.cc:260] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.845793 22143 ts_tablet_manager.cc:1428] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250112 02:57:18.847102 22143 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:18.853785 22144 ts_tablet_manager.cc:1428] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.157s user 0.006s sys 0.000s
I20250112 02:57:18.698532 22386 tablet.cc:1721] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:18.855224 22144 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:18.861857 22389 raft_consensus.cc:2798] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:18.862324 22389 raft_consensus.cc:491] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:18.862803 22389 raft_consensus.cc:3054] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.862962 22143 tablet_bootstrap.cc:654] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.864925 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 (DEFAULT_TABLE table=default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.865931 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7847b7af9e0b4aaba1f7cd06534f6510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.867944 22375 tablet_replica.cc:331] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
W20250112 02:57:18.877228 21383 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bca1c68aae24a1f9974a0bd2244fbf8 consensus state has no leader
I20250112 02:57:18.877256 22375 raft_consensus.cc:2238] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:18.877769 22375 raft_consensus.cc:2267] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:18.878482 21390 master_service.cc:946] ReplaceTablet: received request to replace tablet bb324cd3120e45449e0500c99338135d from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:18.849803 22145 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:18.879863 22145 ts_tablet_manager.cc:1397] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.002s
I20250112 02:57:18.883810 22375 ts_tablet_manager.cc:1905] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:18.891546 22145 raft_consensus.cc:357] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.892472 22145 raft_consensus.cc:383] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:18.896036 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d (DEFAULT_TABLE table=default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.897541 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4998c034e54b1bae0ef132cef5b04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.898417 22144 tablet_bootstrap.cc:654] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:18.903018 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d (DEFAULT_TABLE table=default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.904044 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4998c034e54b1bae0ef132cef5b04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.919235 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_267a17c8b93540128cfb0744673fe49e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:18.920573 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_267a17c8b93540128cfb0744673fe49e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:18.924245 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.925027 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet a6bfe4c753f84aeeba7446def9898d43 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:18.925726 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:18.932349 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0bca1c68aae24a1f9974a0bd2244fbf8 consensus state has no leader
I20250112 02:57:18.933373 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 4166dcd1f9d74255b21810dd42b7358d (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.933373 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 77d80c6a60bc46449c7c0656128100c8 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.934326 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d80c6a60bc46449c7c0656128100c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.934319 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4166dcd1f9d74255b21810dd42b7358d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.934463 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 4166dcd1f9d74255b21810dd42b7358d (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.935384 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4166dcd1f9d74255b21810dd42b7358d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.935182 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 77d80c6a60bc46449c7c0656128100c8 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.943306 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d80c6a60bc46449c7c0656128100c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.943243 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d (DEFAULT_TABLE table=default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.944372 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4998c034e54b1bae0ef132cef5b04d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.949425 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 4166dcd1f9d74255b21810dd42b7358d (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.950569 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4166dcd1f9d74255b21810dd42b7358d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.955331 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 77d80c6a60bc46449c7c0656128100c8 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:18.956516 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77d80c6a60bc46449c7c0656128100c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:18.893563 22389 raft_consensus.cc:513] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:18.969990 22389 leader_election.cc:290] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.971468 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:18.972117 22043 raft_consensus.cc:3054] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.976660 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82d2ac91c5554a64936f364e4e1978f8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:18.977219 21755 raft_consensus.cc:3054] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:18.991328 22250 consensus_queue.cc:1035] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:18.991796 22383 raft_consensus.cc:491] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:18.992210 22383 raft_consensus.cc:513] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:18.993753 21390 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_02f0da840c0646d59412dcb267a4d571" } modify_external_catalogs: true
I20250112 02:57:18.994087 22383 leader_election.cc:290] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:18.995738 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d4e98b6fa4aecb5955e8603eac779" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:18.995693 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d4e98b6fa4aecb5955e8603eac779" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:18.996357 22045 raft_consensus.cc:2463] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:18.997462 21690 leader_election.cc:304] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:19.000025 22383 raft_consensus.cc:2798] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:19.000325 22383 raft_consensus.cc:491] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:19.000626 22383 raft_consensus.cc:3054] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.003196 21755 raft_consensus.cc:2463] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:18.892814 22145 raft_consensus.cc:738] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:19.010669 21691 leader_election.cc:343] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.012405 21835 leader_election.cc:304] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:19.014024 22043 raft_consensus.cc:2463] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:19.015424 22383 raft_consensus.cc:513] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.015959 22145 consensus_queue.cc:260] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.032032 22145 ts_tablet_manager.cc:1428] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.152s user 0.005s sys 0.000s
I20250112 02:57:19.032907 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d4e98b6fa4aecb5955e8603eac779" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:19.033679 22383 leader_election.cc:290] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:19.035265 22376 ts_tablet_manager.cc:1918] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:19.041589 22043 raft_consensus.cc:3054] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.038054 22389 raft_consensus.cc:2798] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:19.044108 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7d4e98b6fa4aecb5955e8603eac779" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:19.046672 21691 leader_election.cc:343] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.043706 22389 raft_consensus.cc:695] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:19.048431 22389 consensus_queue.cc:237] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:19.049171 22406 tablet.cc:1721] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.051842 22143 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:19.052212 22143 ts_tablet_manager.cc:1397] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.205s user 0.022s sys 0.019s
I20250112 02:57:19.037509 22376 log.cc:1198] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:19.059443 22145 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:19.062728 22376 ts_tablet_manager.cc:1939] T 35b39d0cf44f4e91908b8e568df0a16a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:19.054468 22143 raft_consensus.cc:357] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.067844 22143 raft_consensus.cc:383] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.068107 22143 raft_consensus.cc:738] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.068610 22375 ts_tablet_manager.cc:1918] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:19.079155 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:19.080433 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:19.085088 22143 consensus_queue.cc:260] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.093178 22143 ts_tablet_manager.cc:1428] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250112 02:57:19.097441 22143 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:19.096683 22250 raft_consensus.cc:491] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.098330 22250 raft_consensus.cc:513] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.101404 22250 leader_election.cc:290] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.103470 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:19.104151 21757 raft_consensus.cc:2463] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:19.104599 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 35b39d0cf44f4e91908b8e568df0a16a (table default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]) successfully deleted
I20250112 02:57:19.106868 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:19.106911 21978 leader_election.cc:304] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:19.107566 22250 raft_consensus.cc:2798] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:19.107882 22250 raft_consensus.cc:491] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:19.108165 22250 raft_consensus.cc:3054] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.109429 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:19.110481 21388 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 111) took 1139 ms. Trace:
I20250112 02:57:19.110826 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:19.112344 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 7847b7af9e0b4aaba1f7cd06534f6510 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7847b7af9e0b4aaba1f7cd06534f6510 already in progress: creating tablet
I20250112 02:57:19.114763 22250 raft_consensus.cc:513] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.113114 21390 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_02f0da840c0646d59412dcb267a4d571" } modify_external_catalogs: true
I20250112 02:57:19.116813 22376 tablet_replica.cc:331] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:19.117688 22376 raft_consensus.cc:2238] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:19.119441 22376 raft_consensus.cc:2267] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.121330 22376 ts_tablet_manager.cc:1905] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.118587 22250 leader_election.cc:290] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.124269 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_817027c4f6374ecfb969e968afe83e5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.126416 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_817027c4f6374ecfb969e968afe83e5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.129165 21389 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 27234f3e69d544bdb53fb887874c2c0a
I20250112 02:57:19.130815 21389 catalog_manager.cc:6596] ReplaceTablet: tablet 27234f3e69d544bdb53fb887874c2c0a deleted and replaced by tablet 65eb8c99917247edbdbf5de42f3b505c
I20250112 02:57:19.132561 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 27234f3e69d544bdb53fb887874c2c0a with delete_type TABLET_DATA_DELETED (replaced by tablet 65eb8c99917247edbdbf5de42f3b505c) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:19.134538 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_20d7d2b763a4431f864991b5a241d5c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:19.110826 21388 rpcz_store.cc:276] 0112 02:57:17.971048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.372832 (+401784us) service_pool.cc:225] Handling call
0112 02:57:18.487039 (+114207us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:18.487717 (+ 678us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:18.487853 (+ 136us) catalog_manager.cc:2762] Locking tablets
0112 02:57:18.488463 (+ 610us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:18.949574 (+461111us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:19.106676 (+157102us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:19.106790 (+ 114us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:19.110429 (+ 3639us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:18.729138 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:18.729439 (+ 301us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:18.729458 (+ 19us) write_op.cc:435] Acquired schema lock
0112 02:57:18.729470 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:18.729727 (+ 257us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:18.729756 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:18.729776 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:18.729995 (+ 219us) tablet.cc:698] Row locks acquired
0112 02:57:18.730008 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:18.730092 (+ 84us) write_op.cc:270] Start()
0112 02:57:18.730158 (+ 66us) write_op.cc:276] Timestamp: P: 1736650638730119 usec, L: 0
0112 02:57:18.730172 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:18.866806 (+136634us) log.cc:844] Serialized 734 byte log entry
0112 02:57:18.923866 (+ 57060us) op_driver.cc:464] REPLICATION: finished
0112 02:57:18.925417 (+ 1551us) write_op.cc:301] APPLY: starting
0112 02:57:18.925468 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:18.925564 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:18.925647 (+ 83us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:18.925846 (+ 199us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:18.925968 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:18.926004 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:18.926160 (+ 156us) log.cc:844] Serialized 29 byte log entry
0112 02:57:18.945590 (+ 19430us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b6000020dc8. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed9cda81 00007f7eedb37f55 00007f7eedb39f04 00007f7eedaae263 00007f7eedaaf6b6 00007f7eedaaf11c 00007f7eeda9b1b9 00007f7eeda9a983 00007f7eeda9eb91 00007f7eeda9eb49 00007f7eeda9ead9
0112 02:57:18.945634 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.945706 (+ 72us) write_op.cc:454] Released schema lock
0112 02:57:18.945737 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:18.945958 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.945981 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1397,"apply.run_cpu_time_us":2925,"apply.run_wall_time_us":20508,"mutex_wait_us":236300,"num_ops":2,"prepare.queue_time_us":238156,"prepare.run_cpu_time_us":1837,"prepare.run_wall_time_us":158328,"raft.queue_time_us":10762,"raft.run_cpu_time_us":542,"raft.run_wall_time_us":542,"replication_time_us":193589,"spinlock_wait_cycles":40948992}]]}
I20250112 02:57:19.135680 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:19.136197 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20d7d2b763a4431f864991b5a241d5c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.136912 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:19.097918 22375 log.cc:1198] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/64d2be215ded4b8a931339af5ef9b8dd
I20250112 02:57:19.138358 22043 raft_consensus.cc:2463] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:19.138705 22375 ts_tablet_manager.cc:1939] T 64d2be215ded4b8a931339af5ef9b8dd P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:19.141623 22375 tablet_replica.cc:331] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:19.143247 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 64d2be215ded4b8a931339af5ef9b8dd (table default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]) successfully deleted
W20250112 02:57:19.143879 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet cad224c5ad844512b50da5190cfd1861 because the tablet was not found. No further retry: Not found: Tablet not found: cad224c5ad844512b50da5190cfd1861
I20250112 02:57:19.144054 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:19.144443 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet a6bfe4c753f84aeeba7446def9898d43 because the tablet was not found. No further retry: Not found: Tablet not found: a6bfe4c753f84aeeba7446def9898d43
I20250112 02:57:19.144778 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 27234f3e69d544bdb53fb887874c2c0a with delete_type TABLET_DATA_DELETED (replaced by tablet 65eb8c99917247edbdbf5de42f3b505c) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:19.132490 22389 raft_consensus.cc:491] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.145622 22389 raft_consensus.cc:513] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:19.147356 22389 leader_election.cc:290] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:19.151553 21980 leader_election.cc:343] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:19.152375 21980 leader_election.cc:343] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.158865 22145 tablet_bootstrap.cc:654] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.137132 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e3131ae5453b4ce58463562eeb65edec" } new_table_name: "default.Table_07b589dc0b534181bb90a5c327001b87" modify_external_catalogs: true
I20250112 02:57:19.161516 22144 tablet_bootstrap.cc:492] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:19.161916 22144 ts_tablet_manager.cc:1397] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.307s user 0.014s sys 0.007s
I20250112 02:57:19.166016 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 27234f3e69d544bdb53fb887874c2c0a with delete_type TABLET_DATA_DELETED (replaced by tablet 65eb8c99917247edbdbf5de42f3b505c) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.169960 22143 tablet_bootstrap.cc:654] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.170012 22144 raft_consensus.cc:357] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.172678 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:19.173282 21757 raft_consensus.cc:3054] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.170711 22144 raft_consensus.cc:383] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.178915 22144 raft_consensus.cc:738] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.179636 22144 consensus_queue.cc:260] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.180552 22412 raft_consensus.cc:491] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.180917 22412 raft_consensus.cc:513] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.182714 22379 tablet_replica.cc:331] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:19.182814 22412 leader_election.cc:290] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:19.183799 22379 raft_consensus.cc:2238] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:19.184589 22379 raft_consensus.cc:2267] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:19.185950 21835 leader_election.cc:343] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:19.188017 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:19.142355 22375 raft_consensus.cc:2238] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:19.190263 22379 ts_tablet_manager.cc:1905] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:19.190564 21691 leader_election.cc:343] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.194103 22250 consensus_queue.cc:1035] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:57:19.203523 21836 leader_election.cc:343] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:19.203951 21836 leader_election.cc:304] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:19.207392 21757 raft_consensus.cc:2463] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:19.207880 22375 raft_consensus.cc:2267] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.208310 21978 leader_election.cc:304] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:19.209429 22250 raft_consensus.cc:2798] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:19.216392 22375 ts_tablet_manager.cc:1905] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.219790 22389 raft_consensus.cc:2743] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:19.227320 22250 raft_consensus.cc:695] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:19.228312 22250 consensus_queue.cc:237] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.233351 22144 ts_tablet_manager.cc:1428] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.071s user 0.005s sys 0.001s
I20250112 02:57:19.235451 22145 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:19.235831 22145 ts_tablet_manager.cc:1397] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.179s user 0.015s sys 0.001s
W20250112 02:57:19.236603 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 209072us
I20250112 02:57:19.240103 22144 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:19.246366 22145 raft_consensus.cc:357] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.247149 22145 raft_consensus.cc:383] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.247435 22145 raft_consensus.cc:738] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.252768 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:19.253460 22145 consensus_queue.cc:260] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.264768 22145 ts_tablet_manager.cc:1428] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250112 02:57:19.265682 22145 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:19.285290 21690 leader_election.cc:304] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:19.286116 22383 raft_consensus.cc:2798] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:19.286549 22383 raft_consensus.cc:695] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:19.287267 22383 consensus_queue.cc:237] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.297382 22143 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
W20250112 02:57:19.306260 21388 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 has no consensus state
I20250112 02:57:19.306361 22250 raft_consensus.cc:491] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.306990 22250 raft_consensus.cc:513] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.309804 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bca1c68aae24a1f9974a0bd2244fbf8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:19.310148 22143 ts_tablet_manager.cc:1397] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.213s user 0.010s sys 0.007s
W20250112 02:57:19.311070 21980 leader_election.cc:343] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.318118 22385 tablet.cc:1721] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.327215 22406 tablet.cc:1721] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.328804 22145 tablet_bootstrap.cc:654] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:19.303275 21690 leader_election.cc:343] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.331269 21690 leader_election.cc:304] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:19.279628 22144 tablet_bootstrap.cc:654] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.343689 22145 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:19.344059 22145 ts_tablet_manager.cc:1397] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.012s
I20250112 02:57:19.344281 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bca1c68aae24a1f9974a0bd2244fbf8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:19.344910 21757 raft_consensus.cc:2463] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:19.345764 22414 raft_consensus.cc:2743] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:19.345912 21978 leader_election.cc:304] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:19.346206 22145 raft_consensus.cc:357] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.346810 22382 raft_consensus.cc:2798] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:19.346884 22145 raft_consensus.cc:383] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:19.346801 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 43588.9us
I20250112 02:57:19.347149 22382 raft_consensus.cc:491] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:19.347486 22382 raft_consensus.cc:3054] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.348384 22383 consensus_queue.cc:1035] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:19.347163 22145 raft_consensus.cc:738] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:19.349324 21690 consensus_peers.cc:487] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:19.349535 22145 consensus_queue.cc:260] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.353297 22145 ts_tablet_manager.cc:1428] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250112 02:57:19.354017 22145 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:19.355237 22414 consensus_queue.cc:1035] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:19.356446 22382 raft_consensus.cc:513] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.360358 22145 tablet_bootstrap.cc:654] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.361132 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bca1c68aae24a1f9974a0bd2244fbf8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:19.362370 21980 leader_election.cc:343] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.328152 22250 leader_election.cc:290] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.367700 22437 tablet.cc:1721] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.368324 22382 leader_election.cc:290] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.376675 22143 raft_consensus.cc:357] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.377384 22143 raft_consensus.cc:383] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.377264 22383 consensus_queue.cc:1035] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:19.377678 22143 raft_consensus.cc:738] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.379468 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2ae6922ba3dd4bbabb88dd71998911a5" } modify_external_catalogs: true
I20250112 02:57:19.380031 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2ae6922ba3dd4bbabb88dd71998911a5" } modify_external_catalogs: true
I20250112 02:57:19.381731 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_4d7b917596ea4edc81355ff870e26b8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:19.377729 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bca1c68aae24a1f9974a0bd2244fbf8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:19.383076 21756 raft_consensus.cc:3054] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:19.383450 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d7b917596ea4edc81355ff870e26b8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.384370 22144 tablet_bootstrap.cc:492] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:19.384733 22144 ts_tablet_manager.cc:1397] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.145s user 0.004s sys 0.004s
I20250112 02:57:19.385326 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_06528f9a9c8d4cb991d30e195f3944be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.387012 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06528f9a9c8d4cb991d30e195f3944be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.404382 21756 raft_consensus.cc:2463] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:19.425746 21978 leader_election.cc:304] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:19.426107 21392 master_service.cc:946] ReplaceTablet: received request to replace tablet 1e7d4e98b6fa4aecb5955e8603eac779 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:19.418927 22144 raft_consensus.cc:357] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.426563 22144 raft_consensus.cc:383] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.426826 22382 raft_consensus.cc:2798] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:19.426836 22144 raft_consensus.cc:738] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.427237 22382 raft_consensus.cc:695] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:19.427513 22144 consensus_queue.cc:260] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.428012 22382 consensus_queue.cc:237] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.432061 22144 ts_tablet_manager.cc:1428] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20250112 02:57:19.442945 22375 ts_tablet_manager.cc:1918] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:19.443331 22375 log.cc:1198] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/bb324cd3120e45449e0500c99338135d
I20250112 02:57:19.444059 22375 ts_tablet_manager.cc:1939] T bb324cd3120e45449e0500c99338135d P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:19.446601 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet bb324cd3120e45449e0500c99338135d (table default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]) successfully deleted
I20250112 02:57:19.450630 22376 ts_tablet_manager.cc:1918] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:19.451120 22376 log.cc:1198] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/bb324cd3120e45449e0500c99338135d
I20250112 02:57:19.378264 22143 consensus_queue.cc:260] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.466715 22376 ts_tablet_manager.cc:1939] T bb324cd3120e45449e0500c99338135d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:19.470808 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet bb324cd3120e45449e0500c99338135d (table default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]) successfully deleted
I20250112 02:57:19.471235 22376 tablet_replica.cc:331] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:19.472023 22376 raft_consensus.cc:2238] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:19.472326 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet a6bfe4c753f84aeeba7446def9898d43 because the tablet was not found. No further retry: Not found: Tablet not found: a6bfe4c753f84aeeba7446def9898d43
W20250112 02:57:19.473062 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet b1e608afe71349f1ae609e584022e510 because the tablet was not found. No further retry: Not found: Tablet not found: b1e608afe71349f1ae609e584022e510
W20250112 02:57:19.475309 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 49196.4us
I20250112 02:57:19.479480 22144 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:19.484709 22144 tablet_bootstrap.cc:654] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.491423 22375 tablet_replica.cc:331] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
W20250112 02:57:19.491631 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 7847b7af9e0b4aaba1f7cd06534f6510 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7847b7af9e0b4aaba1f7cd06534f6510 already in progress: creating tablet
I20250112 02:57:19.492350 22375 raft_consensus.cc:2238] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:19.493341 22375 raft_consensus.cc:2267] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.494239 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3f830b1a9d5e4df0b421de4dc7d6207b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.495823 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f830b1a9d5e4df0b421de4dc7d6207b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.497889 22376 raft_consensus.cc:2267] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.505731 22375 ts_tablet_manager.cc:1905] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.509038 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_703a5763f259457bad2f7cbc579160be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:19.510001 22376 ts_tablet_manager.cc:1905] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:19.510423 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_703a5763f259457bad2f7cbc579160be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.512334 22143 ts_tablet_manager.cc:1428] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.202s user 0.006s sys 0.000s
I20250112 02:57:19.541121 22376 ts_tablet_manager.cc:1918] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:19.541522 22376 log.cc:1198] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:19.542236 22376 ts_tablet_manager.cc:1939] T 402c24e54ab040e29949b1e4efbe5ff3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:19.547618 22379 ts_tablet_manager.cc:1918] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:19.559504 21385 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b264bbff78dd43b7ac519d9d1b26e30a" } new_table_name: "default.Table_a7196a9422b74e2db6ae571522fa7a2d" modify_external_catalogs: true
I20250112 02:57:19.566661 22379 log.cc:1198] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/27234f3e69d544bdb53fb887874c2c0a
I20250112 02:57:19.567553 22379 ts_tablet_manager.cc:1939] T 27234f3e69d544bdb53fb887874c2c0a P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:19.551538 22143 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:19.568662 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 27234f3e69d544bdb53fb887874c2c0a (table default.Table_432c0b6f219d4c11b784bd6913108e00 [id=a6456005326445aa9356d78a2d599e61]) successfully deleted
I20250112 02:57:19.570987 22375 ts_tablet_manager.cc:1918] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:19.579964 22143 tablet_bootstrap.cc:654] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.588478 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 63) took 1076 ms. Trace:
I20250112 02:57:19.588928 22376 rpcz_store.cc:276] 0112 02:57:18.514136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.514229 (+ 93us) service_pool.cc:225] Handling call
0112 02:57:19.471092 (+956863us) ts_tablet_manager.cc:1145] Deleting tablet 402c24e54ab040e29949b1e4efbe5ff3
0112 02:57:19.471125 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:19.526049 (+ 54924us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:19.540896 (+ 14847us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:19.588410 (+ 47514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956394}
I20250112 02:57:19.590662 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 402c24e54ab040e29949b1e4efbe5ff3 (table default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]) successfully deleted
I20250112 02:57:19.591720 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 57) took 1089 ms. Trace:
I20250112 02:57:19.594668 22375 log.cc:1198] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/27234f3e69d544bdb53fb887874c2c0a
I20250112 02:57:19.590421 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:19.595446 22375 ts_tablet_manager.cc:1939] T 27234f3e69d544bdb53fb887874c2c0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:19.597939 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 5f787ffe65e34e8f9852d149fff9f947 (table default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea [id=e084f29dae5a41899d7cb805c1ea63b6]). Expected version 1 got 0
I20250112 02:57:19.598836 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:19.600481 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 27234f3e69d544bdb53fb887874c2c0a (table default.Table_432c0b6f219d4c11b784bd6913108e00 [id=a6456005326445aa9356d78a2d599e61]) successfully deleted
I20250112 02:57:19.600658 21389 master_service.cc:946] ReplaceTablet: received request to replace tablet 3781431324bc479d97bd22b6bb1243b9 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:19.606958 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 1e7d4e98b6fa4aecb5955e8603eac779
I20250112 02:57:19.608664 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa732d27f144aaad088ecc072837e9) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:19.611429 22376 rpcz_store.cc:276] 0112 02:57:18.502175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.517600 (+ 15425us) service_pool.cc:225] Handling call
0112 02:57:19.591453 (+1073853us) ts_tablet_manager.cc:1145] Deleting tablet 1f77f8b613544e909ed3397c4b2ebd08
0112 02:57:19.591501 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:19.591666 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072673}
I20250112 02:57:19.615211 21392 catalog_manager.cc:6596] ReplaceTablet: tablet 1e7d4e98b6fa4aecb5955e8603eac779 deleted and replaced by tablet 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:19.615655 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa732d27f144aaad088ecc072837e9) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.616465 22379 tablet_replica.cc:331] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:19.616420 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa732d27f144aaad088ecc072837e9) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:19.617172 22375 tablet_replica.cc:331] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:19.617403 22379 raft_consensus.cc:2238] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:19.618115 22379 pending_rounds.cc:62] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Trying to abort 1 pending ops.
I20250112 02:57:19.618091 22375 raft_consensus.cc:2238] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:19.618358 22379 pending_rounds.cc:69] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321018528620544 op_type: NO_OP noop_request { }
I20250112 02:57:19.618629 22375 raft_consensus.cc:2267] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.618834 22379 raft_consensus.cc:2883] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:19.619246 22379 raft_consensus.cc:2267] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.621522 22379 ts_tablet_manager.cc:1905] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.622385 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet afeff74abcc643a5ad6565d9dcc40d62
I20250112 02:57:19.624018 22375 ts_tablet_manager.cc:1905] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.624394 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.624771 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:19.625391 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 1f77f8b613544e909ed3397c4b2ebd08 because the tablet was not found. No further retry: Not found: Tablet not found: 1f77f8b613544e909ed3397c4b2ebd08
I20250112 02:57:19.627283 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:19.628182 22389 raft_consensus.cc:491] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.628640 22389 raft_consensus.cc:513] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.629495 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_15c893814ebd41c2b738d9158e2ae82a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.630952 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15c893814ebd41c2b738d9158e2ae82a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.633001 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:19.633747 21756 raft_consensus.cc:2388] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:19.634680 22376 tablet_replica.cc:331] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:19.635478 22376 raft_consensus.cc:2238] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:19.636294 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33f89398c7aa4f2a938a476db897c833" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:19.637921 21836 leader_election.cc:304] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:19.630422 22389 leader_election.cc:290] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.639096 22389 raft_consensus.cc:3054] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.640864 21619 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet afeff74abcc643a5ad6565d9dcc40d62
I20250112 02:57:19.642500 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:19.644109 22143 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:19.644476 22143 ts_tablet_manager.cc:1397] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.000s
I20250112 02:57:19.645020 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.646703 22022 tablet_service.cc:1467] Processing CreateTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 (DEFAULT_TABLE table=default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.647629 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afeff74abcc643a5ad6565d9dcc40d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.634235 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_99128d2e95814f319baf8037cf642ca5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:19.647734 22389 raft_consensus.cc:2743] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:19.648715 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99128d2e95814f319baf8037cf642ca5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.652240 22376 raft_consensus.cc:2267] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.653720 22143 raft_consensus.cc:357] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.655885 22376 ts_tablet_manager.cc:1905] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.662670 21389 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 3781431324bc479d97bd22b6bb1243b9
I20250112 02:57:19.663924 21389 catalog_manager.cc:6596] ReplaceTablet: tablet 3781431324bc479d97bd22b6bb1243b9 deleted and replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:19.657795 21737 tablet_service.cc:1467] Processing CreateTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 (DEFAULT_TABLE table=default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.667181 21737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afeff74abcc643a5ad6565d9dcc40d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.679203 22143 raft_consensus.cc:383] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.659654 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 99e81199f79d46d6922eb1c6805d25da (DEFAULT_TABLE table=default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.682061 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:19.682952 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e81199f79d46d6922eb1c6805d25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.683847 21878 tablet_service.cc:1467] Processing CreateTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 (DEFAULT_TABLE table=default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.684762 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afeff74abcc643a5ad6565d9dcc40d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.685210 22145 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:19.685564 22145 ts_tablet_manager.cc:1397] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.332s user 0.015s sys 0.012s
I20250112 02:57:19.670850 22382 raft_consensus.cc:491] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.677973 22025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3781431324bc479d97bd22b6bb1243b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:19.678716 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 7af7e02e5fc248839b434f42f95c6509 (DEFAULT_TABLE table=default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.686887 22382 raft_consensus.cc:513] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.687613 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7e02e5fc248839b434f42f95c6509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.689078 22382 leader_election.cc:290] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.691064 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:19.691073 22021 tablet_service.cc:1467] Processing CreateTablet for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c (DEFAULT_TABLE table=default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.691728 22025 tablet_service.cc:1467] Processing CreateTablet for tablet b4dddfea8e8b49189186930fce46ecc5 (DEFAULT_TABLE table=default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.692065 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:19.692248 21980 leader_election.cc:343] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.692724 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dddfea8e8b49189186930fce46ecc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.679764 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 3781431324bc479d97bd22b6bb1243b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.680701 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 99e81199f79d46d6922eb1c6805d25da (DEFAULT_TABLE table=default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.680984 21734 tablet_service.cc:1467] Processing CreateTablet for tablet b4dddfea8e8b49189186930fce46ecc5 (DEFAULT_TABLE table=default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.707736 22144 tablet_bootstrap.cc:492] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:19.681764 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 7af7e02e5fc248839b434f42f95c6509 (DEFAULT_TABLE table=default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.689590 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:19.708109 22144 ts_tablet_manager.cc:1397] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.229s user 0.010s sys 0.009s
I20250112 02:57:19.708611 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7e02e5fc248839b434f42f95c6509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.709494 22143 raft_consensus.cc:738] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.712004 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 99e81199f79d46d6922eb1c6805d25da (DEFAULT_TABLE table=default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.713027 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e81199f79d46d6922eb1c6805d25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.712972 22145 raft_consensus.cc:357] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.713600 22145 raft_consensus.cc:383] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.713822 22145 raft_consensus.cc:738] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.724498 22145 consensus_queue.cc:260] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.722718 22144 raft_consensus.cc:357] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.729810 22144 raft_consensus.cc:383] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:19.730073 22144 raft_consensus.cc:738] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:19.737146 22385 tablet.cc:1721] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.739789 22144 consensus_queue.cc:260] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.740743 22406 tablet.cc:1721] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:19.742238 22145 ts_tablet_manager.cc:1428] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.056s user 0.006s sys 0.001s
I20250112 02:57:19.743247 22145 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:19.766183 22375 ts_tablet_manager.cc:1918] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:19.772637 22022 tablet_service.cc:1467] Processing CreateTablet for tablet eef30508946e4376961b6cd463172c1f (DEFAULT_TABLE table=default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.773185 22020 tablet_service.cc:1467] Processing CreateTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.773600 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef30508946e4376961b6cd463172c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.774338 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d1bc147ba48ebbebfd707dc44fe02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.775327 22021 tablet_service.cc:1467] Processing CreateTablet for tablet 36495fb158a440309eddedd3910a7ad3 (DEFAULT_TABLE table=default.Table_07b589dc0b534181bb90a5c327001b87 [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.776320 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36495fb158a440309eddedd3910a7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.784161 22019 tablet_service.cc:1467] Processing CreateTablet for tablet f47e14720a704a19ac3aea8723e13c28 (DEFAULT_TABLE table=default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.785243 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47e14720a704a19ac3aea8723e13c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.787514 22018 tablet_service.cc:1467] Processing CreateTablet for tablet 22aac4332d0e476d9758004972ef8ab2 (DEFAULT_TABLE table=default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.783231 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3781431324bc479d97bd22b6bb1243b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:19.784309 21881 tablet_service.cc:1467] Processing CreateTablet for tablet b4dddfea8e8b49189186930fce46ecc5 (DEFAULT_TABLE table=default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.788733 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aac4332d0e476d9758004972ef8ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.788777 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dddfea8e8b49189186930fce46ecc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.789347 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 36495fb158a440309eddedd3910a7ad3 (DEFAULT_TABLE table=default.Table_07b589dc0b534181bb90a5c327001b87 [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.790488 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36495fb158a440309eddedd3910a7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:19.793128 22265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:19.793869 22265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250112 02:57:19.785207 21879 tablet_service.cc:1467] Processing CreateTablet for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c (DEFAULT_TABLE table=default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.795004 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.795998 22016 tablet_service.cc:1467] Processing CreateTablet for tablet 557de3de9a4d4d0bac1dc16f66ab6b01 (DEFAULT_TABLE table=default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.796846 21876 tablet_service.cc:1467] Processing CreateTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.797113 22016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557de3de9a4d4d0bac1dc16f66ab6b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.797801 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d1bc147ba48ebbebfd707dc44fe02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.799304 22017 tablet_service.cc:1467] Processing CreateTablet for tablet 21dc0755b38746a39bd773780e48cda5 (DEFAULT_TABLE table=default.Table_2c8b88dcd37b4ecfa44453ce970e61cc [id=c52787432d37482d9ed338ed0ed9e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.800213 22017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dc0755b38746a39bd773780e48cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.803237 21882 tablet_service.cc:1467] Processing CreateTablet for tablet eef30508946e4376961b6cd463172c1f (DEFAULT_TABLE table=default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.804206 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef30508946e4376961b6cd463172c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.786094 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 7af7e02e5fc248839b434f42f95c6509 (DEFAULT_TABLE table=default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.807323 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7af7e02e5fc248839b434f42f95c6509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.807876 21875 tablet_service.cc:1467] Processing CreateTablet for tablet f47e14720a704a19ac3aea8723e13c28 (DEFAULT_TABLE table=default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.808833 21875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47e14720a704a19ac3aea8723e13c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.812304 22389 raft_consensus.cc:491] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.812731 22389 raft_consensus.cc:513] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.814574 22389 leader_election.cc:290] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:19.816262 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 21dc0755b38746a39bd773780e48cda5 (DEFAULT_TABLE table=default.Table_2c8b88dcd37b4ecfa44453ce970e61cc [id=c52787432d37482d9ed338ed0ed9e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.816501 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 557de3de9a4d4d0bac1dc16f66ab6b01 (DEFAULT_TABLE table=default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.817421 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dc0755b38746a39bd773780e48cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.817319 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:19.817879 22043 raft_consensus.cc:2463] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:19.819428 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557de3de9a4d4d0bac1dc16f66ab6b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.822321 21878 tablet_service.cc:1467] Processing CreateTablet for tablet 22aac4332d0e476d9758004972ef8ab2 (DEFAULT_TABLE table=default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.823351 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aac4332d0e476d9758004972ef8ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.823894 21836 leader_election.cc:304] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:19.829903 22265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:57:19.830742 21383 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:19.831017 21733 tablet_service.cc:1467] Processing CreateTablet for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c (DEFAULT_TABLE table=default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.832095 21383 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
W20250112 02:57:19.834110 21378 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 53920.9us
I20250112 02:57:19.835493 21383 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:19.771436 21757 raft_consensus.cc:2463] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:19.848147 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 557de3de9a4d4d0bac1dc16f66ab6b01 (DEFAULT_TABLE table=default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.851522 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:19.852249 21755 raft_consensus.cc:2463] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:19.855835 22144 ts_tablet_manager.cc:1428] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.147s user 0.006s sys 0.000s
I20250112 02:57:19.856849 21731 tablet_service.cc:1467] Processing CreateTablet for tablet eef30508946e4376961b6cd463172c1f (DEFAULT_TABLE table=default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.859166 21730 tablet_service.cc:1467] Processing CreateTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 (DEFAULT_TABLE table=default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.850608 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4dddfea8e8b49189186930fce46ecc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.861902 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 36495fb158a440309eddedd3910a7ad3 (DEFAULT_TABLE table=default.Table_07b589dc0b534181bb90a5c327001b87 [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.866091 21728 tablet_service.cc:1467] Processing CreateTablet for tablet f47e14720a704a19ac3aea8723e13c28 (DEFAULT_TABLE table=default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.870297 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 21dc0755b38746a39bd773780e48cda5 (DEFAULT_TABLE table=default.Table_2c8b88dcd37b4ecfa44453ce970e61cc [id=c52787432d37482d9ed338ed0ed9e682]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.875978 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.879523 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.879725 21727 tablet_service.cc:1467] Processing CreateTablet for tablet 22aac4332d0e476d9758004972ef8ab2 (DEFAULT_TABLE table=default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:19.878237 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_aeb5fca9ba304d2abd6c14dccdf69a16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:19.881026 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:19.891521 21389 catalog_manager.cc:5526] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:19.906033 22450 raft_consensus.cc:2798] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:19.906420 22450 raft_consensus.cc:491] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:19.906770 22450 raft_consensus.cc:3054] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:19.907647 22144 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:19.905614 22375 log.cc:1198] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/1e7d4e98b6fa4aecb5955e8603eac779
I20250112 02:57:19.931998 21389 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5]). Expected version 1 got 0
I20250112 02:57:19.902921 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_04002b2285af49eb9d1fd9edeb48810c" } modify_external_catalogs: true
I20250112 02:57:19.933084 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_04002b2285af49eb9d1fd9edeb48810c" } modify_external_catalogs: true
I20250112 02:57:19.937565 21389 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:19.946877 21389 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:19.949123 22375 ts_tablet_manager.cc:1939] T 1e7d4e98b6fa4aecb5955e8603eac779 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:19.951439 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22aac4332d0e476d9758004972ef8ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.954345 22414 raft_consensus.cc:491] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:19.954732 22414 raft_consensus.cc:513] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.956842 22414 leader_election.cc:290] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:19.956965 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 1e7d4e98b6fa4aecb5955e8603eac779 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:19.957782 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 11fbebff35c344bdb305ea1a6dbbefe4
I20250112 02:57:19.959930 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21dc0755b38746a39bd773780e48cda5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.961153 21724 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:19.962492 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.963109 21389 catalog_manager.cc:5526] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:19.965870 22450 raft_consensus.cc:513] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:19.966858 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eef30508946e4376961b6cd463172c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.967037 22144 tablet_bootstrap.cc:654] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:19.966915 21387 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_8e38f61a93f745f28bf8f5513ba8e4ff [id=682416c8b9f6464789d238eb4e2a1c96] alter complete (version 1)
I20250112 02:57:19.971029 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47e14720a704a19ac3aea8723e13c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.972954 22375 tablet_replica.cc:331] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:19.973744 22375 raft_consensus.cc:2238] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:19.974217 22375 raft_consensus.cc:2267] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:19.975186 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7847b7af9e0b4aaba1f7cd06534f6510" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:19.974745 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afeff74abcc643a5ad6565d9dcc40d62 already in progress: creating tablet
W20250112 02:57:19.976305 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afeff74abcc643a5ad6565d9dcc40d62 already in progress: creating tablet
W20250112 02:57:19.976550 21691 leader_election.cc:343] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:19.978229 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99e81199f79d46d6922eb1c6805d25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.982813 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36495fb158a440309eddedd3910a7ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.986896 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c48d1bc147ba48ebbebfd707dc44fe02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.992796 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:19.993556 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65eb8c99917247edbdbf5de42f3b505c has no consensus state
I20250112 02:57:19.994099 21384 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 119) took 1387 ms. Trace:
I20250112 02:57:19.990970 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557de3de9a4d4d0bac1dc16f66ab6b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:19.994370 21384 rpcz_store.cc:276] 0112 02:57:18.606863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.692270 (+ 85407us) service_pool.cc:225] Handling call
0112 02:57:18.804704 (+112434us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:18.805405 (+ 701us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:18.805968 (+ 563us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:18.806183 (+ 215us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:18.949723 (+143540us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:19.994029 (+1044306us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:18.887598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:18.887962 (+ 364us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:18.887986 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:18.888003 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:18.888254 (+ 251us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:18.888288 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:18.888310 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:18.888441 (+ 131us) tablet.cc:698] Row locks acquired
0112 02:57:18.888454 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:18.888556 (+ 102us) write_op.cc:270] Start()
0112 02:57:18.888632 (+ 76us) write_op.cc:276] Timestamp: P: 1736650638888642 usec, L: 0
0112 02:57:18.888647 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:18.889112 (+ 465us) log.cc:844] Serialized 405 byte log entry
0112 02:57:18.941150 (+ 52038us) op_driver.cc:464] REPLICATION: finished
0112 02:57:18.947564 (+ 6414us) spinlock_profiling.cc:243] Waited 4.13 ms on lock 0x7b6000020dc8. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed9cda81 00007f7eedb3865c 00007f7eeda99c92 00007f7eeda9a2b9 00007f7eeda9c54a 00007f7eeda9c4d8 00007f7eeda9c458 00007f7eeda9c413 00007f7eeda9b62c 00007f7eef38abb6 00007f7eef385568
0112 02:57:18.947848 (+ 284us) write_op.cc:301] APPLY: starting
0112 02:57:18.947891 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:18.947976 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:18.947996 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:18.948138 (+ 142us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:18.948261 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:18.948306 (+ 45us) write_op.cc:312] APPLY: finished
0112 02:57:18.948469 (+ 163us) log.cc:844] Serialized 23 byte log entry
0112 02:57:18.949049 (+ 580us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.949098 (+ 49us) write_op.cc:454] Released schema lock
0112 02:57:18.949128 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:18.949328 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:18.949350 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":1013,"apply.run_wall_time_us":1451,"num_ops":1,"prepare.queue_time_us":6977,"prepare.run_cpu_time_us":1760,"prepare.run_wall_time_us":37555,"replication_time_us":52422,"spinlock_wait_cycles":9496448}]]}
I20250112 02:57:19.998543 22375 ts_tablet_manager.cc:1905] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:19.759704 22143 consensus_queue.cc:260] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.000762 21387 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea [id=e084f29dae5a41899d7cb805c1ea63b6] alter complete (version 1)
I20250112 02:57:20.001663 21390 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:20.002930 21390 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 123) took 1040 ms. Trace:
I20250112 02:57:20.003185 21390 rpcz_store.cc:276] 0112 02:57:18.963552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.968905 (+ 5353us) service_pool.cc:225] Handling call
0112 02:57:18.993655 (+ 24750us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d8616 00007f7eed140b44 00007f7eed140ad2 00007f7eed140a01 00007f7eed14097c 00007f7eed13fc32 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:19.116092 (+122437us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:19.117526 (+ 1434us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:19.117657 (+ 131us) catalog_manager.cc:2762] Locking tablets
0112 02:57:19.118165 (+ 508us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:19.210932 (+ 92767us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:19.211168 (+ 236us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:20.001478 (+790310us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:20.002878 (+ 1400us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:19.155210 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:19.155573 (+ 363us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:19.155602 (+ 29us) write_op.cc:435] Acquired schema lock
0112 02:57:19.155620 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:19.155934 (+ 314us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:19.155979 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:19.156010 (+ 31us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:19.156270 (+ 260us) tablet.cc:698] Row locks acquired
0112 02:57:19.156311 (+ 41us) write_op.cc:260] PREPARE: finished
0112 02:57:19.156438 (+ 127us) write_op.cc:270] Start()
0112 02:57:19.156522 (+ 84us) write_op.cc:276] Timestamp: P: 1736650639156401 usec, L: 0
0112 02:57:19.156542 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 02:57:19.157006 (+ 464us) log.cc:844] Serialized 799 byte log entry
0112 02:57:19.208934 (+ 51928us) op_driver.cc:464] REPLICATION: finished
0112 02:57:19.209288 (+ 354us) write_op.cc:301] APPLY: starting
0112 02:57:19.209356 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:19.209464 (+ 108us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:19.209561 (+ 97us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:19.209782 (+ 221us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:19.209927 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:19.209976 (+ 49us) write_op.cc:312] APPLY: finished
0112 02:57:19.210162 (+ 186us) log.cc:844] Serialized 29 byte log entry
0112 02:57:19.210344 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:19.210419 (+ 75us) write_op.cc:454] Released schema lock
0112 02:57:19.210452 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:19.223300 (+ 12848us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:19.223333 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":53553536,"child_traces":[["op",{"apply.queue_time_us":212,"apply.run_cpu_time_us":1367,"apply.run_wall_time_us":1366,"mutex_wait_us":13688,"num_ops":2,"prepare.queue_time_us":16963,"prepare.run_cpu_time_us":2265,"prepare.run_wall_time_us":67779,"raft.queue_time_us":15669,"raft.run_cpu_time_us":694,"raft.run_wall_time_us":718,"replication_time_us":52296}]]}
I20250112 02:57:20.017095 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:20.018227 22450 leader_election.cc:290] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:20.040843 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 32) took 2142 ms (client timeout 1000 ms). Trace:
W20250112 02:57:20.043151 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_07b589dc0b534181bb90a5c327001b87 [id=55d68af4c9444c96828bd8d85c3bdaf8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
W20250112 02:57:20.041312 21386 rpcz_store.cc:269] 0112 02:57:17.898364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:17.924722 (+ 26358us) service_pool.cc:225] Handling call
0112 02:57:20.040658 (+2115936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32717,"reported_tablets":15}
I20250112 02:57:20.054847 22144 tablet_bootstrap.cc:492] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.055455 22144 ts_tablet_manager.cc:1397] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.148s user 0.006s sys 0.006s
I20250112 02:57:20.061236 21389 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186] alter complete (version 1)
I20250112 02:57:20.072108 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:20.070304 22144 raft_consensus.cc:357] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.075711 22144 raft_consensus.cc:383] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.076002 22144 raft_consensus.cc:738] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.078034 22145 tablet_bootstrap.cc:654] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.083916 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:20.081674 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7847b7af9e0b4aaba1f7cd06534f6510" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:20.096501 22045 raft_consensus.cc:2463] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:20.184804 22144 consensus_queue.cc:260] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.208287 22379 ts_tablet_manager.cc:1918] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:20.208647 22379 log.cc:1198] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/1e7d4e98b6fa4aecb5955e8603eac779
I20250112 02:57:20.209383 22379 ts_tablet_manager.cc:1939] T 1e7d4e98b6fa4aecb5955e8603eac779 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
W20250112 02:57:20.214879 21387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 23) took 1595 ms (client timeout 1000 ms). Trace:
W20250112 02:57:20.215181 21387 rpcz_store.cc:269] 0112 02:57:18.619091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.658065 (+ 38974us) service_pool.cc:225] Handling call
0112 02:57:20.214790 (+1556725us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:19.284132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:19.284482 (+ 350us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:19.284502 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:19.284515 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:19.284731 (+ 216us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:19.284758 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:19.284778 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:19.284886 (+ 108us) tablet.cc:698] Row locks acquired
0112 02:57:19.284896 (+ 10us) write_op.cc:260] PREPARE: finished
0112 02:57:19.285011 (+ 115us) write_op.cc:270] Start()
0112 02:57:19.285075 (+ 64us) write_op.cc:276] Timestamp: P: 1736650639285021 usec, L: 0
0112 02:57:19.285089 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:19.285488 (+ 399us) log.cc:844] Serialized 356 byte log entry
0112 02:57:19.308868 (+ 23380us) op_driver.cc:464] REPLICATION: finished
0112 02:57:19.309039 (+ 171us) write_op.cc:301] APPLY: starting
0112 02:57:19.309074 (+ 35us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:19.309132 (+ 58us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:19.309209 (+ 77us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:19.309321 (+ 112us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:19.309432 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:19.309465 (+ 33us) write_op.cc:312] APPLY: finished
0112 02:57:19.309600 (+ 135us) log.cc:844] Serialized 23 byte log entry
0112 02:57:19.309698 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:19.309744 (+ 46us) write_op.cc:454] Released schema lock
0112 02:57:19.309768 (+ 24us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:19.977330 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:19.977678 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:19.977702 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:19.977715 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:19.977968 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:19.978012 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:19.978035 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:19.978198 (+ 163us) tablet.cc:698] Row locks acquired
0112 02:57:19.978215 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:19.978365 (+ 150us) write_op.cc:270] Start()
0112 02:57:19.978564 (+ 199us) write_op.cc:276] Timestamp: P: 1736650639978444 usec, L: 0
0112 02:57:19.978588 (+ 24us) op_driver.cc:348] REPLICATION: starting
0112 02:57:19.979108 (+ 520us) log.cc:844] Serialized 356 byte log entry
0112 02:57:19.985807 (+ 6699us) op_driver.cc:464] REPLICATION: finished
0112 02:57:19.986068 (+ 261us) write_op.cc:301] APPLY: starting
0112 02:57:19.986118 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:19.986197 (+ 79us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:19.986276 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:19.986435 (+ 159us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:19.986572 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:19.986610 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:19.986756 (+ 146us) log.cc:844] Serialized 23 byte log entry
0112 02:57:19.986896 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:19.986960 (+ 64us) write_op.cc:454] Released schema lock
0112 02:57:19.986985 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:19.987590 (+ 605us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:19.987612 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23758,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":63,"apply.run_cpu_time_us":905,"apply.run_wall_time_us":1526,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":80544,"prepare.run_cpu_time_us":1597,"prepare.run_wall_time_us":1639,"raft.queue_time_us":1849,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":23689}],["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":1089,"apply.run_wall_time_us":1489,"num_ops":1,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":1944,"prepare.run_wall_time_us":2977,"raft.queue_time_us":434,"raft.run_cpu_time_us":594,"raft.run_wall_time_us":594,"replication_time_us":7145}]]}
I20250112 02:57:20.227831 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.234614 22143 ts_tablet_manager.cc:1428] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.590s user 0.010s sys 0.005s
W20250112 02:57:20.234958 22143 ts_tablet_manager.cc:1460] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Tablet startup took 590ms
I20250112 02:57:20.234928 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:20.235289 22143 ts_tablet_manager.cc:1462] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Trace:
0112 02:57:18.902137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.902270 (+ 133us) service_pool.cc:225] Handling call
0112 02:57:18.903636 (+ 1366us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:18.903714 (+ 78us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:18.923232 (+ 19518us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:18.929816 (+ 6584us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:18.930225 (+ 409us) inbound_call.cc:173] Queueing success response
0112 02:57:19.544375 (+614150us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0112 02:57:19.644713 (+100338us) ts_tablet_manager.cc:1429] Starting tablet replica
0112 02:57:19.651576 (+ 6863us) tablet_replica.cc:260] Starting instrumentation
0112 02:57:19.652626 (+ 1050us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":3401,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":157,"tablet-open.queue_time_us":614228}
I20250112 02:57:20.237042 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:20.239296 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:20.240746 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:20.241200 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 119513us
I20250112 02:57:20.241861 21690 leader_election.cc:304] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:20.242305 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 27234f3e69d544bdb53fb887874c2c0a with delete_type TABLET_DATA_DELETED (replaced by tablet 65eb8c99917247edbdbf5de42f3b505c) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:20.243779 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 64d2be215ded4b8a931339af5ef9b8dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:20.245035 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 29) took 1296 ms (client timeout 1000 ms). Trace:
I20250112 02:57:20.245415 22414 raft_consensus.cc:2798] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:20.245762 22414 raft_consensus.cc:491] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:20.246133 22414 raft_consensus.cc:3054] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:20.245313 21391 rpcz_store.cc:269] 0112 02:57:18.948361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.948489 (+ 128us) service_pool.cc:225] Handling call
0112 02:57:18.975451 (+ 26962us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:20.244878 (+1269427us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72017,"reported_tablets":18,"spinlock_wait_cycles":53065984}
I20250112 02:57:20.250149 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.251211 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:20.255558 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:20.257694 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:20.268857 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:20.269403 21755 raft_consensus.cc:3054] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:20.272418 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 33) took 1348 ms (client timeout 1000 ms). Trace:
W20250112 02:57:20.272902 21383 rpcz_store.cc:269] 0112 02:57:18.923570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.959049 (+ 35479us) service_pool.cc:225] Handling call
0112 02:57:18.984149 (+ 25100us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:20.272338 (+1288189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175206,"reported_tablets":20,"spinlock_wait_cycles":27016064}
I20250112 02:57:20.275691 21755 raft_consensus.cc:2463] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:20.280790 22144 ts_tablet_manager.cc:1428] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.225s user 0.003s sys 0.003s
W20250112 02:57:20.293007 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 444634us
I20250112 02:57:20.309561 21901 raft_consensus.cc:3054] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.319576 21901 raft_consensus.cc:1270] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Refusing update from remote peer e0406c08e9714e87bdd7fd6b66c9756d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250112 02:57:20.322201 22018 consensus_meta.cc:287] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.506s user 0.005s sys 0.000s
I20250112 02:57:20.325771 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.327452 21737 tablet_service.cc:1514] Processing DeleteTablet for tablet cad224c5ad844512b50da5190cfd1861 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.328648 22144 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:20.331878 21724 tablet_service.cc:1514] Processing DeleteTablet for tablet 27234f3e69d544bdb53fb887874c2c0a with delete_type TABLET_DATA_DELETED (replaced by tablet 65eb8c99917247edbdbf5de42f3b505c) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.332827 21724 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:20.347594 22406 tablet.cc:1721] T 43c413e1f8fe48039c4dd1b066fdd857 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.322882 22457 consensus_queue.cc:1035] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:20.362998 22375 ts_tablet_manager.cc:1918] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:57:20.363610 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 121893us
I20250112 02:57:20.363106 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 43c413e1f8fe48039c4dd1b066fdd857 (table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143]). Expected version 1 got 0
I20250112 02:57:20.376109 22144 tablet_bootstrap.cc:654] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.386441 22144 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.386852 22144 ts_tablet_manager.cc:1397] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.010s
I20250112 02:57:20.389235 22144 raft_consensus.cc:357] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.389899 22144 raft_consensus.cc:383] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.390141 22144 raft_consensus.cc:738] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.390887 22144 consensus_queue.cc:260] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.389832 22414 raft_consensus.cc:513] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.400013 22414 leader_election.cc:290] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:20.401738 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7847b7af9e0b4aaba1f7cd06534f6510" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:20.408299 22143 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:20.409089 21835 leader_election.cc:304] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:20.410656 22389 raft_consensus.cc:2798] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:20.411489 22450 consensus_queue.cc:1035] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:57:20.416678 21691 consensus_peers.cc:487] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:20.417200 21691 leader_election.cc:343] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:20.419700 22379 tablet_replica.cc:331] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:20.420238 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 1e7d4e98b6fa4aecb5955e8603eac779 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:20.420624 22379 raft_consensus.cc:2238] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:20.420992 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afeff74abcc643a5ad6565d9dcc40d62 already in progress: creating tablet
I20250112 02:57:20.421418 22379 raft_consensus.cc:2267] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.425361 22143 tablet_bootstrap.cc:654] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.429811 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet bb324cd3120e45449e0500c99338135d from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:20.324512 22020 consensus_meta.cc:287] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.542s user 0.005s sys 0.004s
I20250112 02:57:20.331089 22376 ts_tablet_manager.cc:1918] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:20.439121 22376 log.cc:1198] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/64d2be215ded4b8a931339af5ef9b8dd
I20250112 02:57:20.439783 22376 ts_tablet_manager.cc:1939] T 64d2be215ded4b8a931339af5ef9b8dd P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:20.344880 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 270046us
I20250112 02:57:20.350620 22468 tablet.cc:1721] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.441578 21978 leader_election.cc:304] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:20.442063 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afeff74abcc643a5ad6565d9dcc40d62 already in progress: creating tablet
I20250112 02:57:20.442354 22250 raft_consensus.cc:2798] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:20.442688 22250 raft_consensus.cc:491] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:20.443004 22250 raft_consensus.cc:3054] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.446075 22145 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:20.446548 22145 ts_tablet_manager.cc:1397] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.703s user 0.014s sys 0.003s
I20250112 02:57:20.452566 22375 log.cc:1198] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/3781431324bc479d97bd22b6bb1243b9
I20250112 02:57:20.453375 22375 ts_tablet_manager.cc:1939] T 3781431324bc479d97bd22b6bb1243b9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.458153 22375 tablet_replica.cc:331] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:20.458761 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 3781431324bc479d97bd22b6bb1243b9 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:20.462900 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 66) took 1806 ms. Trace:
I20250112 02:57:20.463439 22376 rpcz_store.cc:276] 0112 02:57:18.656521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:18.659736 (+ 3215us) service_pool.cc:225] Handling call
0112 02:57:19.630171 (+970435us) ts_tablet_manager.cc:1145] Deleting tablet 64d2be215ded4b8a931339af5ef9b8dd
0112 02:57:19.634550 (+ 4379us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:19.769983 (+135433us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:20.330863 (+560880us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:20.462828 (+131965us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20817,"tablet-delete.queue_time_us":969910}
I20250112 02:57:20.466121 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 64d2be215ded4b8a931339af5ef9b8dd (table default.Table_7f26edfb6fc44a20b7e131ccbabf6f33 [id=203ba9a0f7c840599ff25b7b53f79405]) successfully deleted
I20250112 02:57:20.468096 22144 ts_tablet_manager.cc:1428] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.081s user 0.005s sys 0.000s
I20250112 02:57:20.469065 22144 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:20.474992 22389 raft_consensus.cc:695] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:20.475911 22389 consensus_queue.cc:237] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.481055 22375 raft_consensus.cc:2238] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.485527 22276 consensus_queue.cc:1035] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:20.487910 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7847b7af9e0b4aaba1f7cd06534f6510" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:20.488438 22043 raft_consensus.cc:3054] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.488060 22144 tablet_bootstrap.cc:654] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.491035 22375 raft_consensus.cc:2267] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.490921 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:20.492336 22470 consensus_queue.cc:1035] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:20.493109 22375 ts_tablet_manager.cc:1905] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.500221 22145 raft_consensus.cc:357] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.501128 22145 raft_consensus.cc:383] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.501457 22145 raft_consensus.cc:738] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.503348 22144 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.503728 22144 ts_tablet_manager.cc:1397] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.006s
I20250112 02:57:20.502774 22145 consensus_queue.cc:260] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.505131 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_d19f37f2c90c4162a8fb2e731a73d7f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:20.506839 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:20.506908 22144 raft_consensus.cc:357] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.507493 22144 raft_consensus.cc:383] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.507726 22144 raft_consensus.cc:738] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.508203 22144 consensus_queue.cc:260] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.511508 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a1ba8d2b677e47f5bffefd5fa58046e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:20.512926 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:20.513135 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea" } modify_external_catalogs: true
I20250112 02:57:20.513617 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea" } modify_external_catalogs: true
I20250112 02:57:20.514676 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a560e6a044654680a00c3488dc51ce17" } new_table_name: "default.Table_f00f4edf62e84429be9874110b97266c" modify_external_catalogs: true
I20250112 02:57:20.515019 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_202c2f292aed4f11a310b5cf5c271aeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:20.516412 22376 tablet_replica.cc:331] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:20.516425 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_202c2f292aed4f11a310b5cf5c271aeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:20.518232 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3299ba87ba964f52b97f94051b2e9431"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:20.519585 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3299ba87ba964f52b97f94051b2e9431 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:20.510587 22144 ts_tablet_manager.cc:1428] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250112 02:57:20.522152 22144 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:20.525234 22145 ts_tablet_manager.cc:1428] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.078s user 0.003s sys 0.002s
I20250112 02:57:20.519310 22250 raft_consensus.cc:513] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.528651 22144 tablet_bootstrap.cc:654] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.545277 22045 raft_consensus.cc:2388] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:20.549423 22250 leader_election.cc:290] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:20.550647 22043 raft_consensus.cc:2463] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:20.551282 22376 raft_consensus.cc:2238] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.552307 22376 raft_consensus.cc:2267] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.555380 22376 ts_tablet_manager.cc:1905] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.558485 22145 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:20.563192 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 50209.9us
I20250112 02:57:20.573139 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:20.576299 22144 tablet_bootstrap.cc:492] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.576685 22144 ts_tablet_manager.cc:1397] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.055s user 0.014s sys 0.026s
I20250112 02:57:20.576699 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_fffe2a1efd5e4ff3973a842a73b42291" } modify_external_catalogs: true
I20250112 02:57:20.577191 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_fffe2a1efd5e4ff3973a842a73b42291" } modify_external_catalogs: true
I20250112 02:57:20.580816 21387 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_07b589dc0b534181bb90a5c327001b87" } new_table_name: "default.Table_c9690c1ccf2845e4a070fb8e6069cd60" modify_external_catalogs: true
I20250112 02:57:20.585492 22375 ts_tablet_manager.cc:1918] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:20.585841 22375 log.cc:1198] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/11fbebff35c344bdb305ea1a6dbbefe4
I20250112 02:57:20.586529 22375 ts_tablet_manager.cc:1939] T 11fbebff35c344bdb305ea1a6dbbefe4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.578749 22144 raft_consensus.cc:357] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.587700 22144 raft_consensus.cc:383] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.587950 22144 raft_consensus.cc:738] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.588485 22144 consensus_queue.cc:260] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.589859 22406 tablet.cc:1721] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.591971 22144 ts_tablet_manager.cc:1428] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20250112 02:57:20.592370 22468 tablet.cc:1721] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.594754 22375 tablet_replica.cc:331] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:20.595391 22375 raft_consensus.cc:2238] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.595889 22379 ts_tablet_manager.cc:1905] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.595865 22375 raft_consensus.cc:2267] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.596326 22144 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:20.600515 22143 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:20.600972 22143 ts_tablet_manager.cc:1397] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.193s user 0.025s sys 0.008s
I20250112 02:57:20.602403 22144 tablet_bootstrap.cc:654] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.604581 22375 ts_tablet_manager.cc:1905] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.607802 22450 raft_consensus.cc:491] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:20.608155 22450 raft_consensus.cc:513] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.610225 22450 leader_election.cc:290] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:20.610675 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4998c034e54b1bae0ef132cef5b04d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:20.612426 21836 leader_election.cc:343] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:20.623126 22375 ts_tablet_manager.cc:1918] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:20.623487 22375 log.cc:1198] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:20.625183 22375 ts_tablet_manager.cc:1939] T 12f556cd7d6f4e13a525d3b8d4207d94 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.626638 22375 tablet_replica.cc:331] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:20.627218 22375 raft_consensus.cc:2238] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.627590 22375 raft_consensus.cc:2267] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.629104 22375 ts_tablet_manager.cc:1905] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.639701 22382 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:20.640123 22382 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.642855 22382 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:20.647931 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:20.650315 21980 leader_election.cc:343] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:20.652153 22145 tablet_bootstrap.cc:654] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.654125 22143 raft_consensus.cc:357] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.654984 22143 raft_consensus.cc:383] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.655344 22143 raft_consensus.cc:738] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.656404 21690 leader_election.cc:304] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:20.656271 22143 consensus_queue.cc:260] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.660274 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4998c034e54b1bae0ef132cef5b04d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:20.660981 21756 raft_consensus.cc:2463] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:20.658463 22375 ts_tablet_manager.cc:1918] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:20.662194 22375 log.cc:1198] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:20.662343 22414 raft_consensus.cc:2798] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:20.662890 22375 ts_tablet_manager.cc:1939] T 5890ebb3c55943d5a334d6c93fbdaf0e P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.662787 22414 raft_consensus.cc:695] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:20.663494 22414 consensus_queue.cc:237] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.667423 22375 tablet_replica.cc:331] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:20.668084 22375 raft_consensus.cc:2238] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.668462 22375 raft_consensus.cc:2267] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.673229 22375 ts_tablet_manager.cc:1905] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.681308 21835 leader_election.cc:304] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters: e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:20.682751 22450 raft_consensus.cc:2798] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:20.683197 22450 raft_consensus.cc:491] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:20.683583 22450 raft_consensus.cc:3054] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.706614 22375 ts_tablet_manager.cc:1918] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:20.706980 22375 log.cc:1198] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:20.707545 22375 ts_tablet_manager.cc:1939] T 35b39d0cf44f4e91908b8e568df0a16a P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.707535 21756 raft_consensus.cc:1270] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Refusing update from remote peer e0406c08e9714e87bdd7fd6b66c9756d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:57:20.711786 22375 tablet_replica.cc:331] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:20.712466 22375 raft_consensus.cc:2238] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.712906 22375 raft_consensus.cc:2267] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.619228 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]) successfully deleted
W20250112 02:57:20.724804 21378 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 105653us
I20250112 02:57:20.725729 22375 ts_tablet_manager.cc:1905] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:20.725837 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_02f0da840c0646d59412dcb267a4d571 [id=ed996933e39d472992d6f351ae51fb39]) successfully deleted
I20250112 02:57:20.726964 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fbe0e20a157479e9160fc6782e3bf37" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:20.727677 21753 raft_consensus.cc:2388] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0406c08e9714e87bdd7fd6b66c9756d in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:20.732270 21386 master_service.cc:946] ReplaceTablet: received request to replace tablet 4166dcd1f9d74255b21810dd42b7358d from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:20.734387 22382 consensus_queue.cc:1035] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:57:20.741346 22382 consensus_queue.cc:1035] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:20.745261 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:20.768836 22486 tablet.cc:1721] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.771154 22143 ts_tablet_manager.cc:1428] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.170s user 0.009s sys 0.000s
I20250112 02:57:20.771800 22375 ts_tablet_manager.cc:1918] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:20.771986 22143 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:20.772130 22375 log.cc:1198] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:20.772903 22375 ts_tablet_manager.cc:1939] T 402c24e54ab040e29949b1e4efbe5ff3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:20.775566 21978 leader_election.cc:304] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:20.776568 22457 raft_consensus.cc:2743] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250112 02:57:20.726436 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet bb324cd3120e45449e0500c99338135d because the tablet was not found. No further retry: Not found: Tablet not found: bb324cd3120e45449e0500c99338135d
W20250112 02:57:20.783082 21378 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 56688.4us
I20250112 02:57:20.783630 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 5890ebb3c55943d5a334d6c93fbdaf0e (table default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]) successfully deleted
I20250112 02:57:20.784142 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 35b39d0cf44f4e91908b8e568df0a16a (table default.Table_ae37aca456fc4eb9a0dd51f067f4b34c [id=718d08673098400fa859dd5ad8ffffc6]) successfully deleted
I20250112 02:57:20.784651 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 402c24e54ab040e29949b1e4efbe5ff3 (table default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]) successfully deleted
W20250112 02:57:20.785214 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 27234f3e69d544bdb53fb887874c2c0a because the tablet was not found. No further retry: Not found: Tablet not found: 27234f3e69d544bdb53fb887874c2c0a
W20250112 02:57:20.785755 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 64d2be215ded4b8a931339af5ef9b8dd because the tablet was not found. No further retry: Not found: Tablet not found: 64d2be215ded4b8a931339af5ef9b8dd
W20250112 02:57:20.792171 21978 leader_election.cc:343] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:20.792671 21978 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:20.794914 22382 raft_consensus.cc:2743] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:20.804152 21691 consensus_peers.cc:487] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11fbebff35c344bdb305ea1a6dbbefe4. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:20.807077 21691 consensus_peers.cc:487] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:20.819205 22145 tablet_bootstrap.cc:492] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:20.819881 22145 ts_tablet_manager.cc:1397] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.262s user 0.013s sys 0.000s
I20250112 02:57:20.824132 22144 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.824506 22144 ts_tablet_manager.cc:1397] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.228s user 0.025s sys 0.078s
I20250112 02:57:20.838323 22450 raft_consensus.cc:513] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.841476 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4998c034e54b1bae0ef132cef5b04d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:20.842905 22144 raft_consensus.cc:357] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.839358 22495 tablet.cc:1721] T 43c413e1f8fe48039c4dd1b066fdd857 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.822333 22145 raft_consensus.cc:357] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.843595 22144 raft_consensus.cc:383] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.849928 22144 raft_consensus.cc:738] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.850400 22145 raft_consensus.cc:383] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.850525 22144 consensus_queue.cc:260] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.852567 22144 ts_tablet_manager.cc:1428] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.027s user 0.004s sys 0.001s
I20250112 02:57:20.853034 22145 raft_consensus.cc:738] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.853442 22144 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:20.853572 22490 tablet.cc:1721] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:20.856833 22145 consensus_queue.cc:260] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.858744 21391 master_service.cc:946] ReplaceTablet: received request to replace tablet f9bb62ab3f274c29b2c5f9528890b816 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:20.861848 22450 leader_election.cc:290] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:20.862733 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4998c034e54b1bae0ef132cef5b04d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:20.863288 21755 raft_consensus.cc:3054] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.868201 22044 raft_consensus.cc:3054] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:20.873256 22145 ts_tablet_manager.cc:1428] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.053s user 0.006s sys 0.000s
I20250112 02:57:20.874040 22145 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:20.874442 22143 tablet_bootstrap.cc:654] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.877465 22144 tablet_bootstrap.cc:654] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:20.883627 22265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:57:20.883989 22265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:20.884313 22265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:57:20.884605 22265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:57:20.888717 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3cc36619a0114526980225e5fb00e357"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:20.890094 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3cc36619a0114526980225e5fb00e357 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:20.920296 22144 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:20.920751 22144 ts_tablet_manager.cc:1397] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.067s user 0.013s sys 0.007s
I20250112 02:57:20.927831 22376 ts_tablet_manager.cc:1918] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:20.928229 22376 log.cc:1198] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:20.928885 22376 ts_tablet_manager.cc:1939] T 7847b7af9e0b4aaba1f7cd06534f6510 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:20.930707 22145 tablet_bootstrap.cc:654] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:20.931524 22044 raft_consensus.cc:2463] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:20.933192 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 72) took 1824 ms. Trace:
I20250112 02:57:20.933292 21836 leader_election.cc:304] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:20.934073 22450 raft_consensus.cc:2798] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:20.933550 22376 rpcz_store.cc:276] 0112 02:57:19.109144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.109265 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:20.516249 (+1406984us) ts_tablet_manager.cc:1145] Deleting tablet 7847b7af9e0b4aaba1f7cd06534f6510
0112 02:57:20.516295 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:20.552107 (+ 35812us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b54000e38d4. stack: 00007f9555e2c853 00007f9555e2c750 00007f95559a1584 00000000004fcd24 00000000004fccb9 00007f955cd2c6d1 00007f955cdaf352 00007f955d1cc1ac 00007f955dddcd5a 00007f955de0cca0 00007f955ddf3e6d 00007f955ddf3e19 00007f955ddf3da9 00007f955ddf3d71 00007f955ddf2f9d 000000000059ea04
0112 02:57:20.714573 (+162466us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:20.927559 (+212986us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:20.933119 (+ 5560us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78250752,"tablet-delete.queue_time_us":1406391}
I20250112 02:57:20.934474 22450 raft_consensus.cc:695] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:20.937112 22450 consensus_queue.cc:237] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.940387 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 7847b7af9e0b4aaba1f7cd06534f6510 (table default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]) successfully deleted
I20250112 02:57:20.941149 22376 tablet_replica.cc:331] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:20.942112 22376 raft_consensus.cc:2238] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:20.942711 22376 raft_consensus.cc:2267] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:20.944787 22376 ts_tablet_manager.cc:1905] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:20.984112 21387 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
I20250112 02:57:20.984753 22145 tablet_bootstrap.cc:492] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:20.985150 22145 ts_tablet_manager.cc:1397] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.111s user 0.014s sys 0.040s
I20250112 02:57:20.987425 22145 raft_consensus.cc:357] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.987697 21755 raft_consensus.cc:2463] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:20.988081 22145 raft_consensus.cc:383] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:20.988318 22145 raft_consensus.cc:738] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:20.988857 22145 consensus_queue.cc:260] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.996255 22143 tablet_bootstrap.cc:492] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:20.996693 22143 ts_tablet_manager.cc:1397] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.225s user 0.009s sys 0.002s
I20250112 02:57:20.999027 22143 raft_consensus.cc:357] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:20.999796 22143 raft_consensus.cc:383] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.000092 22143 raft_consensus.cc:738] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.000830 22143 consensus_queue.cc:260] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.002645 22379 ts_tablet_manager.cc:1918] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:21.003006 22379 log.cc:1198] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/3781431324bc479d97bd22b6bb1243b9
I20250112 02:57:21.005034 22143 ts_tablet_manager.cc:1428] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 02:57:21.007864 22145 ts_tablet_manager.cc:1428] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250112 02:57:21.008667 22145 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:21.015821 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b5f650c3424f49d995b8bf5cef1f146e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:21.017383 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5f650c3424f49d995b8bf5cef1f146e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.022791 22145 tablet_bootstrap.cc:654] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:21.023294 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet f9bb62ab3f274c29b2c5f9528890b816
I20250112 02:57:21.024628 21391 catalog_manager.cc:6596] ReplaceTablet: tablet f9bb62ab3f274c29b2c5f9528890b816 deleted and replaced by tablet e8f7301bed6c4d98a80c12b70073e62f
I20250112 02:57:21.027597 21878 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:21.028201 22375 tablet_replica.cc:331] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:21.028823 22375 raft_consensus.cc:2238] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:21.029207 22375 raft_consensus.cc:2267] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.031080 22375 ts_tablet_manager.cc:1905] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.034775 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:21.043350 21728 tablet_service.cc:1467] Processing CreateTablet for tablet e0977a10252840a381234f02bb7339b3 (DEFAULT_TABLE table=default.Table_3f830b1a9d5e4df0b421de4dc7d6207b [id=9864c693b5b044c3b2317b83da0b0064]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.044674 21878 tablet_service.cc:1467] Processing CreateTablet for tablet e0977a10252840a381234f02bb7339b3 (DEFAULT_TABLE table=default.Table_3f830b1a9d5e4df0b421de4dc7d6207b [id=9864c693b5b044c3b2317b83da0b0064]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.045969 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (DEFAULT_TABLE table=default.Table_06528f9a9c8d4cb991d30e195f3944be [id=2fa952f6cb4e4fd5a40b77000c30d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.055792 22379 ts_tablet_manager.cc:1939] T 3781431324bc479d97bd22b6bb1243b9 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:21.064814 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_24f9f706cc244d319aa60e1f687d5325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:21.066489 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24f9f706cc244d319aa60e1f687d5325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.075704 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (DEFAULT_TABLE table=default.Table_06528f9a9c8d4cb991d30e195f3944be [id=2fa952f6cb4e4fd5a40b77000c30d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.077803 21876 tablet_service.cc:1467] Processing CreateTablet for tablet f08c1affbaf247059ab65222e2e95bab (DEFAULT_TABLE table=default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.082679 21727 tablet_service.cc:1467] Processing CreateTablet for tablet f08c1affbaf247059ab65222e2e95bab (DEFAULT_TABLE table=default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.084746 21390 catalog_manager.cc:5526] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:21.088284 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 1f21e3c769364b58bd371aac338db1f2 (DEFAULT_TABLE table=default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.091289 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 1f21e3c769364b58bd371aac338db1f2 (DEFAULT_TABLE table=default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.092483 22143 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:21.093663 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_c9690c1ccf2845e4a070fb8e6069cd60 [id=55d68af4c9444c96828bd8d85c3bdaf8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
I20250112 02:57:21.096237 21882 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bf172abad43ac9a519310334d1640 (DEFAULT_TABLE table=default.Table_267a17c8b93540128cfb0744673fe49e [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.099143 21390 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 43c413e1f8fe48039c4dd1b066fdd857 (table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143]). Expected version 1 got 0
I20250112 02:57:21.101202 21390 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:21.102634 22144 raft_consensus.cc:357] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.104938 22265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:34893,127.20.217.189:36049,127.20.217.188:35683
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:36049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.217.189
--webserver_port=45653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:21.106647 21730 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bf172abad43ac9a519310334d1640 (DEFAULT_TABLE table=default.Table_267a17c8b93540128cfb0744673fe49e [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.115284 22265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:21.109627 21881 tablet_service.cc:1467] Processing CreateTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (DEFAULT_TABLE table=default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.115985 22144 raft_consensus.cc:383] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.116235 22144 raft_consensus.cc:738] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.117563 22265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:57:21.119055 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (DEFAULT_TABLE table=default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.122028 22450 raft_consensus.cc:491] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.122443 22450 raft_consensus.cc:513] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.124804 22450 leader_election.cc:290] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.131554 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08c1affbaf247059ab65222e2e95bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.131409 22144 consensus_queue.cc:260] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.132882 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f21e3c769364b58bd371aac338db1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.132171 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7847b7af9e0b4aaba1f7cd06534f6510" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:21.135068 21878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0977a10252840a381234f02bb7339b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.136220 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.137296 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8bf172abad43ac9a519310334d1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.138332 21881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f0d6a5afd140a8af77a5a39dc39dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.141719 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa732d27f144aaad088ecc072837e9 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.142763 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa732d27f144aaad088ecc072837e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.153478 21731 tablet_service.cc:1467] Processing CreateTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.154453 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9543a0a0359e4b72bd30f1cb6290f767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.159942 21875 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa732d27f144aaad088ecc072837e9 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.161054 21875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa732d27f144aaad088ecc072837e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.178746 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:21.180088 21386 catalog_manager.cc:6596] ReplaceTablet: tablet 4166dcd1f9d74255b21810dd42b7358d deleted and replaced by tablet e21a0e4b5ebc426383cf332f46492b80
I20250112 02:57:21.181825 21733 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166dcd1f9d74255b21810dd42b7358d with delete_type TABLET_DATA_DELETED (replaced by tablet e21a0e4b5ebc426383cf332f46492b80) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:21.183440 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166dcd1f9d74255b21810dd42b7358d with delete_type TABLET_DATA_DELETED (replaced by tablet e21a0e4b5ebc426383cf332f46492b80) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:21.195634 22144 ts_tablet_manager.cc:1428] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.275s user 0.001s sys 0.005s
W20250112 02:57:21.199785 22510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:21.203878 21379 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 94988.9us
I20250112 02:57:21.206907 22143 tablet_bootstrap.cc:654] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:21.209381 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250112 02:57:21.211238 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.212329 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9543a0a0359e4b72bd30f1cb6290f767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.212661 21733 tablet_service.cc:1467] Processing CreateTablet for tablet 65eb8c99917247edbdbf5de42f3b505c (DEFAULT_TABLE table=default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.215236 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 65eb8c99917247edbdbf5de42f3b505c (DEFAULT_TABLE table=default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.216153 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eb8c99917247edbdbf5de42f3b505c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.217993 22450 raft_consensus.cc:491] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.218390 22450 raft_consensus.cc:513] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.220153 22450 leader_election.cc:290] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.222169 22144 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:21.231184 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166dcd1f9d74255b21810dd42b7358d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:21.231848 21756 raft_consensus.cc:2463] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:21.232807 21835 leader_election.cc:304] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:21.233601 22450 raft_consensus.cc:2798] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:21.233909 22450 raft_consensus.cc:491] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:21.234208 22450 raft_consensus.cc:3054] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.239271 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3350597bcb8d4f3191e9c6ebed009ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:21.240761 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3350597bcb8d4f3191e9c6ebed009ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.241789 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_8e38f61a93f745f28bf8f5513ba8e4ff" } new_table_name: "default.Table_dc2380b6e80d458d8260d4bc07ded652" modify_external_catalogs: true
I20250112 02:57:21.243185 21873 tablet_service.cc:1467] Processing CreateTablet for tablet c80b7813b50d4739a76776d5d16434f3 (DEFAULT_TABLE table=default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.244710 21735 tablet_service.cc:1467] Processing CreateTablet for tablet c80b7813b50d4739a76776d5d16434f3 (DEFAULT_TABLE table=default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.252749 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 1800d0eb7a9a48d295aca16607291add (DEFAULT_TABLE table=default.Table_703a5763f259457bad2f7cbc579160be [id=96df1b8c0560443981f1df06a75cc9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.255066 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 1800d0eb7a9a48d295aca16607291add (DEFAULT_TABLE table=default.Table_703a5763f259457bad2f7cbc579160be [id=96df1b8c0560443981f1df06a75cc9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.268829 22389 raft_consensus.cc:491] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.269268 22389 raft_consensus.cc:513] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.271229 22389 leader_election.cc:290] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.273000 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:21.274255 21835 leader_election.cc:343] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:21.305817 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166dcd1f9d74255b21810dd42b7358d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:21.306845 22043 raft_consensus.cc:2463] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
W20250112 02:57:21.309820 21836 leader_election.cc:343] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:21.310210 21836 leader_election.cc:304] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:21.311355 22389 raft_consensus.cc:2743] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:21.312533 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:21.314062 21836 leader_election.cc:343] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:21.314558 21836 leader_election.cc:304] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:21.315276 22389 raft_consensus.cc:2743] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:21.325811 22118 log.cc:926] Time spent T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20250112 02:57:21.326822 22114 log.cc:926] Time spent T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250112 02:57:21.330019 22375 ts_tablet_manager.cc:1918] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:21.330363 22375 log.cc:1198] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/f9bb62ab3f274c29b2c5f9528890b816
I20250112 02:57:21.335417 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80b7813b50d4739a76776d5d16434f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.341462 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1800d0eb7a9a48d295aca16607291add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.349887 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0977a10252840a381234f02bb7339b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.350561 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8bf172abad43ac9a519310334d1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.351053 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.354736 22144 tablet_bootstrap.cc:654] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:21.355489 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08c1affbaf247059ab65222e2e95bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.356446 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f21e3c769364b58bd371aac338db1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.359889 22143 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:21.360255 22143 ts_tablet_manager.cc:1397] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.353s user 0.009s sys 0.002s
I20250112 02:57:21.362767 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80b7813b50d4739a76776d5d16434f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.363718 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f0d6a5afd140a8af77a5a39dc39dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.368428 22375 ts_tablet_manager.cc:1939] T f9bb62ab3f274c29b2c5f9528890b816 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:21.368511 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eb8c99917247edbdbf5de42f3b505c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.373889 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1800d0eb7a9a48d295aca16607291add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.378713 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 77) took 1699 ms. Trace:
I20250112 02:57:21.379079 22379 rpcz_store.cc:276] 0112 02:57:19.679532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.679639 (+ 107us) service_pool.cc:225] Handling call
0112 02:57:20.419559 (+739920us) ts_tablet_manager.cc:1145] Deleting tablet 3781431324bc479d97bd22b6bb1243b9
0112 02:57:20.419595 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:20.827815 (+408220us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.002365 (+174550us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.378652 (+376287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":711923}
I20250112 02:57:21.380195 22379 tablet_replica.cc:331] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:21.380970 22379 raft_consensus.cc:2238] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:21.381680 22379 raft_consensus.cc:2267] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.384588 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 3781431324bc479d97bd22b6bb1243b9 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:21.392154 22376 ts_tablet_manager.cc:1918] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:21.392848 22376 log.cc:1198] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/27234f3e69d544bdb53fb887874c2c0a
I20250112 02:57:21.393877 22376 ts_tablet_manager.cc:1939] T 27234f3e69d544bdb53fb887874c2c0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:21.403472 22379 ts_tablet_manager.cc:1905] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.408402 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_aa7c0fcb28e54a549bbc2c16662666a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:21.409881 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa7c0fcb28e54a549bbc2c16662666a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.412129 22145 tablet_bootstrap.cc:492] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:21.412621 22145 ts_tablet_manager.cc:1397] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.404s user 0.021s sys 0.027s
I20250112 02:57:21.423733 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:21.425194 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.432245 22143 raft_consensus.cc:357] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.433491 22143 raft_consensus.cc:383] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.433969 22143 raft_consensus.cc:738] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.434681 22143 consensus_queue.cc:260] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.438081 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 73) took 2305 ms. Trace:
I20250112 02:57:21.446305 22375 tablet_replica.cc:331] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:21.465816 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet f9bb62ab3f274c29b2c5f9528890b816 (table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]) successfully deleted
I20250112 02:57:21.509557 22389 consensus_queue.cc:1035] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:21.513124 22144 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:21.513353 22389 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.513518 22144 ts_tablet_manager.cc:1397] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.292s user 0.010s sys 0.001s
I20250112 02:57:21.513720 22389 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.516394 22389 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.499130 22461 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.526748 22461 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.528642 22461 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:21.517007 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:21.529582 21756 raft_consensus.cc:2463] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:21.530661 22414 raft_consensus.cc:491] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:21.531035 22414 raft_consensus.cc:513] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.532171 21835 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:21.532907 22389 raft_consensus.cc:2798] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:21.533201 22389 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:21.533510 22389 raft_consensus.cc:3054] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.534325 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:21.538007 22414 leader_election.cc:290] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:21.538874 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:21.539633 21901 raft_consensus.cc:2463] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:21.541224 21691 leader_election.cc:304] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:21.542225 22414 raft_consensus.cc:2798] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:21.542750 22414 raft_consensus.cc:491] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:21.543088 22414 raft_consensus.cc:3054] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.560207 22450 raft_consensus.cc:513] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.561208 22375 raft_consensus.cc:2238] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:21.563437 22375 raft_consensus.cc:2267] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.568821 22450 leader_election.cc:290] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.574287 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166dcd1f9d74255b21810dd42b7358d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:21.574883 21756 raft_consensus.cc:3054] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.574988 22375 ts_tablet_manager.cc:1905] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.440580 22145 raft_consensus.cc:357] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.592075 22145 raft_consensus.cc:383] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.592434 22145 raft_consensus.cc:738] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.591998 22376 rpcz_store.cc:276] 0112 02:57:19.132197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.132333 (+ 136us) service_pool.cc:225] Handling call
0112 02:57:20.940116 (+1807783us) ts_tablet_manager.cc:1145] Deleting tablet 27234f3e69d544bdb53fb887874c2c0a
0112 02:57:20.941019 (+ 903us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:21.029796 (+ 88777us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.341906 (+312110us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.437970 (+ 96064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806898}
I20250112 02:57:21.593046 22145 consensus_queue.cc:260] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.593690 22376 tablet_replica.cc:331] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:21.445235 22020 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:21.594702 22376 raft_consensus.cc:2238] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:21.448647 22018 tablet_service.cc:1467] Processing CreateTablet for tablet e0977a10252840a381234f02bb7339b3 (DEFAULT_TABLE table=default.Table_3f830b1a9d5e4df0b421de4dc7d6207b [id=9864c693b5b044c3b2317b83da0b0064]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.596195 22018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0977a10252840a381234f02bb7339b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.595258 22376 raft_consensus.cc:2267] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.450326 22019 tablet_service.cc:1467] Processing CreateTablet for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (DEFAULT_TABLE table=default.Table_06528f9a9c8d4cb991d30e195f3944be [id=2fa952f6cb4e4fd5a40b77000c30d75e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.598807 22376 ts_tablet_manager.cc:1905] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.451927 22016 tablet_service.cc:1467] Processing CreateTablet for tablet f08c1affbaf247059ab65222e2e95bab (DEFAULT_TABLE table=default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.599720 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4166dcd1f9d74255b21810dd42b7358d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:21.600217 22045 raft_consensus.cc:3054] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.453949 22021 tablet_service.cc:1467] Processing CreateTablet for tablet 1f21e3c769364b58bd371aac338db1f2 (DEFAULT_TABLE table=default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.459326 22022 tablet_service.cc:1467] Processing CreateTablet for tablet cf8bf172abad43ac9a519310334d1640 (DEFAULT_TABLE table=default.Table_267a17c8b93540128cfb0744673fe49e [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.460755 22017 tablet_service.cc:1467] Processing CreateTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (DEFAULT_TABLE table=default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.462118 22025 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa732d27f144aaad088ecc072837e9 (DEFAULT_TABLE table=default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.477435 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166dcd1f9d74255b21810dd42b7358d with delete_type TABLET_DATA_DELETED (replaced by tablet e21a0e4b5ebc426383cf332f46492b80) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:21.479123 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 (DEFAULT_TABLE table=default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.480583 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 65eb8c99917247edbdbf5de42f3b505c (DEFAULT_TABLE table=default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.482106 22014 tablet_service.cc:1467] Processing CreateTablet for tablet c80b7813b50d4739a76776d5d16434f3 (DEFAULT_TABLE table=default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.483709 22013 tablet_service.cc:1467] Processing CreateTablet for tablet 1800d0eb7a9a48d295aca16607291add (DEFAULT_TABLE table=default.Table_703a5763f259457bad2f7cbc579160be [id=96df1b8c0560443981f1df06a75cc9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.520280 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:21.604306 22044 raft_consensus.cc:2463] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:21.531520 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:21.605028 22043 raft_consensus.cc:2463] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:21.605674 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 27234f3e69d544bdb53fb887874c2c0a (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]) successfully deleted
I20250112 02:57:21.536033 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:21.608726 22042 raft_consensus.cc:2463] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:21.611579 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 402c24e54ab040e29949b1e4efbe5ff3 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:21.619037 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.515772 22144 raft_consensus.cc:357] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.621819 22144 raft_consensus.cc:383] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.622084 22144 raft_consensus.cc:738] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.623138 22016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08c1affbaf247059ab65222e2e95bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.627148 22021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f21e3c769364b58bd371aac338db1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.630183 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf8bf172abad43ac9a519310334d1640. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.630075 22144 consensus_queue.cc:260] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.634982 22017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27f0d6a5afd140a8af77a5a39dc39dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.639067 22025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98aa732d27f144aaad088ecc072837e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.643808 22144 ts_tablet_manager.cc:1428] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.130s user 0.006s sys 0.000s
I20250112 02:57:21.646044 22024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9543a0a0359e4b72bd30f1cb6290f767. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.655428 21690 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:21.657835 22461 raft_consensus.cc:2798] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:21.658149 22461 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:21.658564 22461 raft_consensus.cc:3054] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.666815 22389 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.669168 22389 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:21.670655 21902 raft_consensus.cc:2388] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:21.673853 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:21.678440 21756 raft_consensus.cc:2463] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:21.679037 22524 consensus_queue.cc:1035] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:21.681526 21835 leader_election.cc:304] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
W20250112 02:57:21.682132 21835 raft_consensus.cc:2681] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:57:21.686691 22144 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:21.687037 21691 consensus_peers.cc:487] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12f556cd7d6f4e13a525d3b8d4207d94. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:21.713366 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80b7813b50d4739a76776d5d16434f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.715821 22144 tablet_bootstrap.cc:654] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:21.719065 22013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1800d0eb7a9a48d295aca16607291add. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.726830 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:21.727371 22042 raft_consensus.cc:3054] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.738758 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65eb8c99917247edbdbf5de42f3b505c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.761031 22414 raft_consensus.cc:513] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.762967 22414 leader_election.cc:290] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:21.764179 22379 ts_tablet_manager.cc:1918] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:21.764524 22379 log.cc:1198] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/11fbebff35c344bdb305ea1a6dbbefe4
I20250112 02:57:21.765206 22379 ts_tablet_manager.cc:1939] T 11fbebff35c344bdb305ea1a6dbbefe4 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:21.766712 22145 ts_tablet_manager.cc:1428] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.354s user 0.007s sys 0.002s
I20250112 02:57:21.767534 22145 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:21.769134 22143 ts_tablet_manager.cc:1428] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.409s user 0.010s sys 0.000s
I20250112 02:57:21.771768 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:21.772333 21901 raft_consensus.cc:3054] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.774978 21902 raft_consensus.cc:3054] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.790145 22375 ts_tablet_manager.cc:1918] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:21.790603 22375 log.cc:1198] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:21.791289 22375 ts_tablet_manager.cc:1939] T 4166dcd1f9d74255b21810dd42b7358d P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:21.795455 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afeff74abcc643a5ad6565d9dcc40d62" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:21.795994 22044 raft_consensus.cc:3054] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:21.799574 22143 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:21.834093 21388 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5]). Expected version 1 got 0
I20250112 02:57:21.835079 21388 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]). Expected version 1 got 0
I20250112 02:57:21.836136 21388 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 43c413e1f8fe48039c4dd1b066fdd857 (table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143]). Expected version 1 got 0
I20250112 02:57:21.836678 21388 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:21.837442 22414 raft_consensus.cc:491] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e0406c08e9714e87bdd7fd6b66c9756d)
I20250112 02:57:21.837903 22414 raft_consensus.cc:513] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:21.840395 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:21.841714 21392 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 148) took 1270 ms. Trace:
I20250112 02:57:21.842713 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:21.843806 22023 tablet_service.cc:1514] Processing DeleteTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:21.843569 22414 leader_election.cc:290] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:21.844537 21691 leader_election.cc:343] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Not found: Tablet not found: bb324cd3120e45449e0500c99338135d
I20250112 02:57:21.844959 22145 tablet_bootstrap.cc:654] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:21.845249 21690 leader_election.cc:343] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: bb324cd3120e45449e0500c99338135d
I20250112 02:57:21.841979 21392 rpcz_store.cc:276] 0112 02:57:20.571630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.571761 (+ 131us) service_pool.cc:225] Handling call
0112 02:57:20.576611 (+ 4850us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d8616 00007f7eed140b44 00007f7eed140ad2 00007f7eed140a01 00007f7eed14097c 00007f7eed13fc32 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:20.577533 (+ 922us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:20.577974 (+ 441us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:20.578071 (+ 97us) catalog_manager.cc:2762] Locking tablets
0112 02:57:21.051345 (+473274us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:21.184543 (+133198us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:21.184721 (+ 178us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:21.840305 (+655584us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:21.841660 (+ 1355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:21.052995 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:21.053298 (+ 303us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:21.053318 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:21.053330 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:21.053602 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:21.053643 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:21.053663 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:21.053913 (+ 250us) tablet.cc:698] Row locks acquired
0112 02:57:21.053926 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:21.054022 (+ 96us) write_op.cc:270] Start()
0112 02:57:21.054094 (+ 72us) write_op.cc:276] Timestamp: P: 1736650641053736 usec, L: 0
0112 02:57:21.054110 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:21.054594 (+ 484us) log.cc:844] Serialized 734 byte log entry
0112 02:57:21.066125 (+ 11531us) spinlock_profiling.cc:243] Waited 9.52 ms on lock 0x7b58000f02b0. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed5aba94 00007f7eed5b09d1 00007f7eed5b0959 00007f7eed5b08e9 00007f7eed5b08b1 00007f7eed5afa7d 000000000059ea04 000000000059e839 00007f7ee66e9156 00007f7ee66ec5b1
0112 02:57:21.172218 (+106093us) op_driver.cc:464] REPLICATION: finished
0112 02:57:21.173882 (+ 1664us) write_op.cc:301] APPLY: starting
0112 02:57:21.173930 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:21.174024 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:21.174039 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:21.174293 (+ 254us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:21.174419 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:21.174464 (+ 45us) write_op.cc:312] APPLY: finished
0112 02:57:21.183789 (+ 9325us) log.cc:844] Serialized 29 byte log entry
0112 02:57:21.183921 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:21.183984 (+ 63us) write_op.cc:454] Released schema lock
0112 02:57:21.184020 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:21.184226 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:21.184248 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":9324288,"child_traces":[["op",{"apply.queue_time_us":1542,"apply.run_cpu_time_us":1197,"apply.run_wall_time_us":10313,"num_ops":2,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":1820,"prepare.run_wall_time_us":1839,"raft.queue_time_us":1363,"raft.run_cpu_time_us":613,"raft.run_wall_time_us":10027,"replication_time_us":118037,"spinlock_wait_cycles":21888256}]]}
I20250112 02:57:21.845631 21690 leader_election.cc:304] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:21.846988 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:21.847570 21392 master_service.cc:946] ReplaceTablet: received request to replace tablet 33a5c293bbd643aab9409c6e9abe4f0a from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:21.851047 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_bd88e7e1e2c549bf8c4f1c74509c4a2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:21.852715 22143 tablet_bootstrap.cc:654] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:21.852715 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd88e7e1e2c549bf8c4f1c74509c4a2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.857301 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 89) took 1897 ms. Trace:
I20250112 02:57:21.857889 22045 raft_consensus.cc:2463] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:21.865422 22379 rpcz_store.cc:276] 0112 02:57:19.959513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.959719 (+ 206us) service_pool.cc:225] Handling call
0112 02:57:21.380045 (+1420326us) ts_tablet_manager.cc:1145] Deleting tablet 11fbebff35c344bdb305ea1a6dbbefe4
0112 02:57:21.380084 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:21.662246 (+282162us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.763935 (+101689us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.857236 (+ 93301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418494}
I20250112 02:57:21.866830 22379 tablet_replica.cc:331] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:21.867842 22379 raft_consensus.cc:2238] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:21.868664 22379 raft_consensus.cc:2267] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.870306 22379 ts_tablet_manager.cc:1905] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.870811 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]) successfully deleted
I20250112 02:57:21.880578 22145 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:21.880656 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8273b629804f1c86a813b04747a1aa (DEFAULT_TABLE table=default.Table_3cc36619a0114526980225e5fb00e357 [id=9b8bea48def14faeb9293473dc2c83a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.880924 22145 ts_tablet_manager.cc:1397] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.002s
I20250112 02:57:21.881597 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8273b629804f1c86a813b04747a1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.885011 22145 raft_consensus.cc:357] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.885637 22145 raft_consensus.cc:383] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.885864 22145 raft_consensus.cc:738] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:21.886435 22145 consensus_queue.cc:260] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.889730 22145 ts_tablet_manager.cc:1428] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250112 02:57:21.892130 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8273b629804f1c86a813b04747a1aa (DEFAULT_TABLE table=default.Table_3cc36619a0114526980225e5fb00e357 [id=9b8bea48def14faeb9293473dc2c83a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.896746 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 994e1d114e5a4de5aa1d7651907d718e (DEFAULT_TABLE table=default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.897733 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994e1d114e5a4de5aa1d7651907d718e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.880162 22533 raft_consensus.cc:2743] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:21.894433 22528 consensus_queue.cc:1035] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:21.916054 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 4166dcd1f9d74255b21810dd42b7358d (table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]) successfully deleted
W20250112 02:57:21.916817 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet c48d1bc147ba48ebbebfd707dc44fe02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48d1bc147ba48ebbebfd707dc44fe02 already in progress: creating tablet
I20250112 02:57:21.890113 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_812849daab4f4ed5b4b29a84dc3dd026"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:21.920034 21389 catalog_manager.cc:5526] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:21.922744 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:21.925046 21392 catalog_manager.cc:6596] ReplaceTablet: tablet 33a5c293bbd643aab9409c6e9abe4f0a deleted and replaced by tablet 46f8052651fa496cbb3702c17f592106
I20250112 02:57:21.925642 22020 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:21.924640 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:21.927266 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_582c4ae70bd94161908109db890c099e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:21.928740 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:21.929405 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 9a8273b629804f1c86a813b04747a1aa (DEFAULT_TABLE table=default.Table_3cc36619a0114526980225e5fb00e357 [id=9b8bea48def14faeb9293473dc2c83a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.929672 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:21.930148 22375 tablet_replica.cc:331] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:21.930840 22375 raft_consensus.cc:2238] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:21.931285 22375 raft_consensus.cc:2267] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:21.930814 21389 catalog_manager.cc:5526] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:21.932967 22375 ts_tablet_manager.cc:1905] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:21.932157 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:21.933774 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:21.935144 22144 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:21.935510 22144 ts_tablet_manager.cc:1397] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.249s user 0.020s sys 0.008s
I20250112 02:57:21.937530 22144 raft_consensus.cc:357] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:21.938134 22144 raft_consensus.cc:383] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:21.939126 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_77fe18e65c724b129a710891e2e7e123"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:21.943902 22145 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:21.951740 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_812849daab4f4ed5b4b29a84dc3dd026 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.939320 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 994e1d114e5a4de5aa1d7651907d718e (DEFAULT_TABLE table=default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.941565 22461 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:21.951040 21690 consensus_peers.cc:487] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7847b7af9e0b4aaba1f7cd06534f6510. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:21.953730 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77fe18e65c724b129a710891e2e7e123 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.955410 21755 raft_consensus.cc:2388] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:21.959407 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:21.960410 21899 raft_consensus.cc:2388] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
W20250112 02:57:21.962482 21980 consensus_peers.cc:487] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:21.963223 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8273b629804f1c86a813b04747a1aa. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:21.966408 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_582c4ae70bd94161908109db890c099e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:21.966912 22044 raft_consensus.cc:2463] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:21.970356 22376 ts_tablet_manager.cc:1918] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:21.970678 22376 log.cc:1198] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/1e7d4e98b6fa4aecb5955e8603eac779
I20250112 02:57:21.971238 22376 ts_tablet_manager.cc:1939] T 1e7d4e98b6fa4aecb5955e8603eac779 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:21.971345 21690 leader_election.cc:304] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:21.960278 22524 consensus_queue.cc:1035] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:21.973816 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:21.975229 21385 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 145) took 1463 ms. Trace:
I20250112 02:57:21.975939 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:21.976748 22020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:21.975522 21385 rpcz_store.cc:276] 0112 02:57:20.511764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.512753 (+ 989us) service_pool.cc:225] Handling call
0112 02:57:20.514000 (+ 1247us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:20.525478 (+ 11478us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:20.525596 (+ 118us) catalog_manager.cc:2762] Locking tablets
0112 02:57:20.526219 (+ 623us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:20.818532 (+292313us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:20.818725 (+ 193us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:21.973730 (+1155005us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:21.975171 (+ 1441us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:20.576130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:20.576482 (+ 352us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:20.576503 (+ 21us) write_op.cc:435] Acquired schema lock
0112 02:57:20.576515 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:20.576804 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:20.576846 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:20.576865 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:20.577087 (+ 222us) tablet.cc:698] Row locks acquired
0112 02:57:20.577097 (+ 10us) write_op.cc:260] PREPARE: finished
0112 02:57:20.577190 (+ 93us) write_op.cc:270] Start()
0112 02:57:20.577260 (+ 70us) write_op.cc:276] Timestamp: P: 1736650640577281 usec, L: 0
0112 02:57:20.577273 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:20.577688 (+ 415us) log.cc:844] Serialized 774 byte log entry
0112 02:57:20.815765 (+238077us) op_driver.cc:464] REPLICATION: finished
0112 02:57:20.816519 (+ 754us) write_op.cc:301] APPLY: starting
0112 02:57:20.816576 (+ 57us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:20.816673 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:20.816774 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:20.817012 (+ 238us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:20.817132 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:20.817171 (+ 39us) write_op.cc:312] APPLY: finished
0112 02:57:20.817336 (+ 165us) log.cc:844] Serialized 29 byte log entry
0112 02:57:20.817468 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.817549 (+ 81us) write_op.cc:454] Released schema lock
0112 02:57:20.817577 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:20.817783 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.817816 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":608,"apply.run_cpu_time_us":1232,"apply.run_wall_time_us":1231,"num_ops":2,"prepare.queue_time_us":48328,"prepare.run_cpu_time_us":1783,"prepare.run_wall_time_us":1783,"raft.queue_time_us":2073,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":16,"replication_time_us":238410}]]}
I20250112 02:57:21.979333 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.979414 22020 tablet_service.cc:1467] Processing CreateTablet for tablet 994e1d114e5a4de5aa1d7651907d718e (DEFAULT_TABLE table=default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:21.980268 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f98f6840e24f3c9d03937c1bbc00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.980394 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994e1d114e5a4de5aa1d7651907d718e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:21.981048 21901 raft_consensus.cc:2463] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:21.990821 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.009466 21727 tablet_service.cc:1467] Processing CreateTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.010558 22533 raft_consensus.cc:2798] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:22.011714 22533 raft_consensus.cc:695] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:22.012037 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.013118 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f98f6840e24f3c9d03937c1bbc00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.012739 22533 consensus_queue.cc:237] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.016757 21873 tablet_service.cc:1467] Processing CreateTablet for tablet f663e183c2ed49b98327b2ad0de12ed9 (DEFAULT_TABLE table=default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.017829 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f663e183c2ed49b98327b2ad0de12ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.017741 22543 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:22.019750 22544 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:21.938364 22144 raft_consensus.cc:738] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.027834 22144 consensus_queue.cc:260] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.040128 22144 ts_tablet_manager.cc:1428] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.104s user 0.005s sys 0.000s
I20250112 02:57:21.959522 22145 tablet_bootstrap.cc:654] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.049409 22389 consensus_queue.cc:1035] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:22.049865 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f2f1590c7a40439c89ff74db3d19cdcf" } modify_external_catalogs: true
I20250112 02:57:22.050403 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_f2f1590c7a40439c89ff74db3d19cdcf" } modify_external_catalogs: true
I20250112 02:57:22.051623 22144 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:22.064862 22144 tablet_bootstrap.cc:654] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.066387 21391 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:22.069231 21391 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:22.069705 22145 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:22.070048 22145 ts_tablet_manager.cc:1397] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.126s user 0.009s sys 0.000s
I20250112 02:57:22.071691 21384 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 146) took 1559 ms. Trace:
I20250112 02:57:22.071951 21383 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602] alter complete (version 1)
W20250112 02:57:22.073516 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 25) took 1767 ms (client timeout 1000 ms). Trace:
I20250112 02:57:22.072573 22145 raft_consensus.cc:357] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.074136 22145 raft_consensus.cc:383] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.074352 22145 raft_consensus.cc:738] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.075028 22145 consensus_queue.cc:260] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.071985 21384 rpcz_store.cc:276] 0112 02:57:20.512034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.514344 (+ 2310us) service_pool.cc:225] Handling call
0112 02:57:20.515292 (+ 948us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:20.516053 (+ 761us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:20.516551 (+ 498us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:20.516747 (+ 196us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:20.731219 (+214472us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:22.071637 (+1340418us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:20.541396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:20.541857 (+ 461us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:20.541892 (+ 35us) write_op.cc:435] Acquired schema lock
0112 02:57:20.541911 (+ 19us) tablet.cc:659] Decoding operations
0112 02:57:20.542257 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:20.542311 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:20.542344 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:20.542549 (+ 205us) tablet.cc:698] Row locks acquired
0112 02:57:20.542570 (+ 21us) write_op.cc:260] PREPARE: finished
0112 02:57:20.542699 (+ 129us) write_op.cc:270] Start()
0112 02:57:20.542801 (+ 102us) write_op.cc:276] Timestamp: P: 1736650640542767 usec, L: 0
0112 02:57:20.542824 (+ 23us) op_driver.cc:348] REPLICATION: starting
0112 02:57:20.543445 (+ 621us) log.cc:844] Serialized 405 byte log entry
0112 02:57:20.727793 (+184348us) op_driver.cc:464] REPLICATION: finished
0112 02:57:20.728450 (+ 657us) write_op.cc:301] APPLY: starting
0112 02:57:20.728504 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:20.728587 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:20.728680 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:20.728802 (+ 122us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:20.728917 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:20.728956 (+ 39us) write_op.cc:312] APPLY: finished
0112 02:57:20.729108 (+ 152us) log.cc:844] Serialized 23 byte log entry
0112 02:57:20.729244 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.729296 (+ 52us) write_op.cc:454] Released schema lock
0112 02:57:20.729324 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:20.729513 (+ 189us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.729534 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":521,"apply.run_cpu_time_us":1036,"apply.run_wall_time_us":1035,"num_ops":1,"prepare.queue_time_us":20871,"prepare.run_cpu_time_us":2366,"prepare.run_wall_time_us":2541,"raft.queue_time_us":90,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":19,"replication_time_us":184880}]]}
I20250112 02:57:22.077227 22145 ts_tablet_manager.cc:1428] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250112 02:57:22.078106 22145 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:22.078639 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:22.085486 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:22.090236 22389 raft_consensus.cc:491] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:22.090677 22389 raft_consensus.cc:513] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.092561 22389 leader_election.cc:290] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:22.094660 22145 tablet_bootstrap.cc:654] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.094894 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:22.095492 22044 raft_consensus.cc:2463] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:22.097019 21836 leader_election.cc:304] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:22.014734 21736 tablet_service.cc:1467] Processing CreateTablet for tablet f663e183c2ed49b98327b2ad0de12ed9 (DEFAULT_TABLE table=default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.093518 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:22.098865 21835 leader_election.cc:343] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:22.073863 21383 rpcz_store.cc:269] 0112 02:57:20.306544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.336229 (+ 29685us) service_pool.cc:225] Handling call
0112 02:57:22.073452 (+1737223us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:20.370420 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:20.370858 (+ 438us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:20.370882 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:20.370897 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:20.371147 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:20.371187 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:20.371210 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:20.371343 (+ 133us) tablet.cc:698] Row locks acquired
0112 02:57:20.371357 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:20.371459 (+ 102us) write_op.cc:270] Start()
0112 02:57:20.371537 (+ 78us) write_op.cc:276] Timestamp: P: 1736650640371464 usec, L: 0
0112 02:57:20.371552 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:20.371990 (+ 438us) log.cc:844] Serialized 356 byte log entry
0112 02:57:20.379640 (+ 7650us) op_driver.cc:464] REPLICATION: finished
0112 02:57:20.380100 (+ 460us) write_op.cc:301] APPLY: starting
0112 02:57:20.380167 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:20.380284 (+ 117us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:20.380391 (+ 107us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:20.380595 (+ 204us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:20.380759 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:20.380819 (+ 60us) write_op.cc:312] APPLY: finished
0112 02:57:20.381029 (+ 210us) log.cc:844] Serialized 23 byte log entry
0112 02:57:20.381184 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.381264 (+ 80us) write_op.cc:454] Released schema lock
0112 02:57:20.381301 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:20.381545 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:20.381578 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":277,"apply.run_cpu_time_us":1403,"apply.run_wall_time_us":1402,"num_ops":1,"prepare.queue_time_us":101,"prepare.run_cpu_time_us":1875,"prepare.run_wall_time_us":1880,"raft.queue_time_us":2944,"raft.run_cpu_time_us":648,"raft.run_wall_time_us":647,"replication_time_us":7978}]]}
W20250112 02:57:22.103960 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 30) took 2085 ms (client timeout 1000 ms). Trace:
I20250112 02:57:22.105631 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:22.106989 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:22.104638 21390 rpcz_store.cc:269] 0112 02:57:20.018853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.019064 (+ 211us) service_pool.cc:225] Handling call
0112 02:57:22.103896 (+2084832us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:21.257652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:21.257994 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:21.258014 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:21.258025 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:21.258275 (+ 250us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:21.258309 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:21.258332 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:21.258476 (+ 144us) tablet.cc:698] Row locks acquired
0112 02:57:21.258492 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:21.258637 (+ 145us) write_op.cc:270] Start()
0112 02:57:21.258705 (+ 68us) write_op.cc:276] Timestamp: P: 1736650641258433 usec, L: 0
0112 02:57:21.258719 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:21.259118 (+ 399us) log.cc:844] Serialized 536 byte log entry
0112 02:57:21.396770 (+137652us) op_driver.cc:464] REPLICATION: finished
0112 02:57:21.397596 (+ 826us) write_op.cc:301] APPLY: starting
0112 02:57:21.397650 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:21.397745 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:21.397867 (+ 122us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:21.398047 (+ 180us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:21.398185 (+ 138us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:21.398229 (+ 44us) write_op.cc:312] APPLY: finished
0112 02:57:21.398405 (+ 176us) log.cc:844] Serialized 23 byte log entry
0112 02:57:21.398559 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:21.398613 (+ 54us) write_op.cc:454] Released schema lock
0112 02:57:21.398644 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:21.398843 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:21.398871 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":197760,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":677,"apply.run_cpu_time_us":1221,"apply.run_wall_time_us":1220,"num_ops":1,"prepare.queue_time_us":147455,"prepare.run_cpu_time_us":1670,"prepare.run_wall_time_us":1820,"raft.queue_time_us":67,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":17,"replication_time_us":137962}]]}
I20250112 02:57:22.108561 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3781431324bc479d97bd22b6bb1243b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:22.108943 22145 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:22.109310 22145 ts_tablet_manager.cc:1397] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.008s
I20250112 02:57:22.110071 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b39d0cf44f4e91908b8e568df0a16a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:22.111367 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:22.111747 22145 raft_consensus.cc:357] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.112401 22145 raft_consensus.cc:383] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.112640 22145 raft_consensus.cc:738] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.113288 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fbebff35c344bdb305ea1a6dbbefe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.113334 22145 consensus_queue.cc:260] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.113806 22389 raft_consensus.cc:2798] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:22.114359 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 3781431324bc479d97bd22b6bb1243b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 9543a0a0359e4b72bd30f1cb6290f767) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.114274 22389 raft_consensus.cc:491] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:22.115192 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7d4e98b6fa4aecb5955e8603eac779 with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa732d27f144aaad088ecc072837e9) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.115401 22389 raft_consensus.cc:3054] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:22.116012 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.119228 22145 ts_tablet_manager.cc:1428] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20250112 02:57:22.121731 21388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 34) took 1463 ms (client timeout 1000 ms). Trace:
W20250112 02:57:22.122640 22545 log.cc:926] Time spent T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
W20250112 02:57:22.125617 21388 rpcz_store.cc:269] 0112 02:57:20.657960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.658116 (+ 156us) service_pool.cc:225] Handling call
0112 02:57:22.121630 (+1463514us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":21}
I20250112 02:57:22.127238 22145 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:22.137243 22145 tablet_bootstrap.cc:654] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.137511 22009 tablet_service.cc:1467] Processing CreateTablet for tablet f663e183c2ed49b98327b2ad0de12ed9 (DEFAULT_TABLE table=default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.138796 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3890251e644c1eb596de043b63491e (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.139669 22010 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3890251e644c1eb596de043b63491e (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.151734 21388 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_c18efd91f8804ad69b7532dbba118163" } new_table_name: "default.Table_4150ff9a4dc84f6e979e362dbcd443b7" modify_external_catalogs: true
I20250112 02:57:22.154300 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_102d4e698b854093aa5caa6c1e25ad50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:22.161339 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_102d4e698b854093aa5caa6c1e25ad50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:22.164903 22556 tablet.cc:1721] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:22.136318 22144 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:22.169936 22461 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:22.170109 22144 ts_tablet_manager.cc:1397] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.004s
I20250112 02:57:22.171013 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d80c6a60bc46449c7c0656128100c8" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:22.171824 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 5e3890251e644c1eb596de043b63491e (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.172575 21690 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:22.174120 22461 raft_consensus.cc:2743] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:57:22.175917 22144 raft_consensus.cc:357] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.177134 22144 raft_consensus.cc:383] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.177372 22144 raft_consensus.cc:738] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.177898 22144 consensus_queue.cc:260] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.167346 22557 tablet.cc:1721] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:22.186000 22144 ts_tablet_manager.cc:1428] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.013s user 0.003s sys 0.003s
I20250112 02:57:22.189862 22144 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:22.191128 22145 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:22.191540 22145 ts_tablet_manager.cc:1397] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.005s
I20250112 02:57:22.194733 21390 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_bd88e7e1e2c549bf8c4f1c74509c4a2a" } modify_external_catalogs: true
I20250112 02:57:22.195243 22144 tablet_bootstrap.cc:654] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.195276 21390 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_bd88e7e1e2c549bf8c4f1c74509c4a2a" } modify_external_catalogs: true
I20250112 02:57:22.202040 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_7833e03b6e984109905db04edeabfa29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:22.203893 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7833e03b6e984109905db04edeabfa29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:22.204111 22145 raft_consensus.cc:357] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.204771 22145 raft_consensus.cc:383] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.205029 22145 raft_consensus.cc:738] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:22.209242 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 65eb8c99917247edbdbf5de42f3b505c consensus state has no leader
I20250112 02:57:22.210763 22145 consensus_queue.cc:260] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.215674 22145 ts_tablet_manager.cc:1428] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.024s user 0.004s sys 0.002s
I20250112 02:57:22.220357 22533 raft_consensus.cc:491] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e0406c08e9714e87bdd7fd6b66c9756d)
I20250112 02:57:22.220862 22533 raft_consensus.cc:513] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:22.223070 22145 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:22.226588 21756 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.020s sys 0.020s
W20250112 02:57:22.226938 21756 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.520s user 0.020s sys 0.020s
I20250112 02:57:22.236246 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 0df734ba87264abd803237a5bbb32da2 (DEFAULT_TABLE table=default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.236817 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 0df734ba87264abd803237a5bbb32da2 (DEFAULT_TABLE table=default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.243520 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 67260685978247898ad224128ed8964f (DEFAULT_TABLE table=default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.247912 22143 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:22.248521 22143 ts_tablet_manager.cc:1397] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.449s user 0.007s sys 0.002s
I20250112 02:57:22.250708 22143 raft_consensus.cc:357] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.251408 22143 raft_consensus.cc:383] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.266361 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3890251e644c1eb596de043b63491e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:22.269315 22042 consensus_meta.cc:287] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.542s user 0.006s sys 0.000s
I20250112 02:57:22.269599 22042 raft_consensus.cc:2463] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:22.266021 22145 tablet_bootstrap.cc:654] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.262960 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f663e183c2ed49b98327b2ad0de12ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.278790 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67260685978247898ad224128ed8964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.279444 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3890251e644c1eb596de043b63491e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.279289 22389 raft_consensus.cc:513] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.280543 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994e1d114e5a4de5aa1d7651907d718e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.281414 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df734ba87264abd803237a5bbb32da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.282528 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:22.283126 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f98f6840e24f3c9d03937c1bbc00e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.288043 22533 leader_election.cc:290] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:22.288833 21835 leader_election.cc:343] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:22.291582 21836 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:22.292392 22524 raft_consensus.cc:2798] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:22.292817 22524 raft_consensus.cc:695] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:22.293512 22524 consensus_queue.cc:237] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.289392 22389 leader_election.cc:290] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:22.297586 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df734ba87264abd803237a5bbb32da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:22.304286 21690 leader_election.cc:343] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:22.271262 22143 raft_consensus.cc:738] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.310976 22143 consensus_queue.cc:260] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.313791 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb (DEFAULT_TABLE table=default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:22.317343 21691 leader_election.cc:343] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Not found: Tablet not found: 402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:22.317956 21691 leader_election.cc:304] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:22.319100 22533 raft_consensus.cc:2743] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:22.328181 22143 ts_tablet_manager.cc:1428] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.079s user 0.005s sys 0.000s
I20250112 02:57:22.329047 22143 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:22.329094 21733 tablet_service.cc:1467] Processing CreateTablet for tablet d6ca275360ec41ceb0f7ef9bfaedf647 (DEFAULT_TABLE table=default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.330264 22524 raft_consensus.cc:491] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:22.330783 22524 raft_consensus.cc:513] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.333302 22524 leader_election.cc:290] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:22.334348 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:22.335927 21835 leader_election.cc:343] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:22.349603 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 84f7700b8cce4261a32f2009d1481b99 (DEFAULT_TABLE table=default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.375304 21731 tablet_service.cc:1467] Processing CreateTablet for tablet e8f7301bed6c4d98a80c12b70073e62f (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.385233 21724 tablet_service.cc:1467] Processing CreateTablet for tablet e21a0e4b5ebc426383cf332f46492b80 (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.381649 22143 tablet_bootstrap.cc:654] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.417119 22143 tablet_bootstrap.cc:492] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:22.417553 22143 ts_tablet_manager.cc:1397] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.000s
I20250112 02:57:22.422205 22143 raft_consensus.cc:357] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.441025 22143 raft_consensus.cc:383] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.441350 22143 raft_consensus.cc:738] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.444715 22143 consensus_queue.cc:260] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.451470 22143 ts_tablet_manager.cc:1428] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.031s user 0.005s sys 0.000s
I20250112 02:57:22.463148 22143 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:22.342443 22025 consensus_meta.cc:287] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.548s user 0.004s sys 0.004s
I20250112 02:57:22.351979 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:22.433671 21980 consensus_peers.cc:487] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:22.483338 22044 raft_consensus.cc:2463] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:22.424476 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 75) took 2816 ms. Trace:
W20250112 02:57:22.483464 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 50090.6us
I20250112 02:57:22.346735 22009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f663e183c2ed49b98327b2ad0de12ed9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:22.421710 22024 consensus_meta.cc:287] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.633s user 0.004s sys 0.000s
I20250112 02:57:22.483994 22376 rpcz_store.cc:276] 0112 02:57:19.608401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.608553 (+ 152us) service_pool.cc:225] Handling call
0112 02:57:21.593465 (+1984912us) ts_tablet_manager.cc:1145] Deleting tablet 1e7d4e98b6fa4aecb5955e8603eac779
0112 02:57:21.593551 (+ 86us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:21.865180 (+271629us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:21.970183 (+105003us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.424414 (+454231us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1982104}
I20250112 02:57:22.485051 22024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 114) took 1007 ms. Trace:
I20250112 02:57:22.485244 22376 tablet_replica.cc:331] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:22.485378 22024 rpcz_store.cc:276] 0112 02:57:21.478137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.478432 (+ 295us) service_pool.cc:225] Handling call
0112 02:57:21.645625 (+167193us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.645708 (+ 83us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.779269 (+133561us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.484534 (+705265us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.485008 (+ 474us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:22.485904 22376 raft_consensus.cc:2238] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:22.486287 22376 raft_consensus.cc:2267] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:22.486572 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e3890251e644c1eb596de043b63491e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.487883 22376 ts_tablet_manager.cc:1905] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:22.393514 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 67260685978247898ad224128ed8964f (DEFAULT_TABLE table=default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.496090 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67260685978247898ad224128ed8964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.496546 22143 tablet_bootstrap.cc:654] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.499362 22533 raft_consensus.cc:491] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:22.404367 22021 consensus_meta.cc:287] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.000s
I20250112 02:57:22.499826 22533 raft_consensus.cc:513] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.501509 22533 leader_election.cc:290] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:22.387548 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb (DEFAULT_TABLE table=default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.505779 22022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.389904 22023 tablet_service.cc:1467] Processing CreateTablet for tablet d6ca275360ec41ceb0f7ef9bfaedf647 (DEFAULT_TABLE table=default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.509193 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ca275360ec41ceb0f7ef9bfaedf647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.513424 22143 tablet_bootstrap.cc:492] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:22.429919 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:22.513808 22143 ts_tablet_manager.cc:1397] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.004s
I20250112 02:57:22.513875 22042 raft_consensus.cc:3054] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:22.483517 22025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 112) took 1022 ms. Trace:
I20250112 02:57:22.516052 22143 raft_consensus.cc:357] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.516117 22025 rpcz_store.cc:276] 0112 02:57:21.461388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.461513 (+ 125us) service_pool.cc:225] Handling call
0112 02:57:21.638555 (+177042us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.638666 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.794198 (+155532us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.483123 (+688925us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.483470 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:22.516644 22143 raft_consensus.cc:383] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.516877 22143 raft_consensus.cc:738] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.437458 22006 tablet_service.cc:1467] Processing CreateTablet for tablet e8f7301bed6c4d98a80c12b70073e62f (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.526129 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a8273b629804f1c86a813b04747a1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.535246 21724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e21a0e4b5ebc426383cf332f46492b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.538214 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.539287 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ca275360ec41ceb0f7ef9bfaedf647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.540447 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f7700b8cce4261a32f2009d1481b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.541930 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8f7301bed6c4d98a80c12b70073e62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.548435 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:22.465134 22016 consensus_meta.cc:287] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.645s user 0.005s sys 0.000s
I20250112 02:57:22.500643 22021 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 109) took 1048 ms. Trace:
W20250112 02:57:22.384603 22013 consensus_meta.cc:287] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.532s user 0.005s sys 0.000s
I20250112 02:57:22.435966 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 84f7700b8cce4261a32f2009d1481b99 (DEFAULT_TABLE table=default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.556265 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:22.558566 21690 leader_election.cc:343] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:22.529095 22143 consensus_queue.cc:260] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.554240 22021 rpcz_store.cc:276] 0112 02:57:21.452640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.453235 (+ 595us) service_pool.cc:225] Handling call
0112 02:57:21.626561 (+173326us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.626701 (+ 140us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.797907 (+171206us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.500144 (+702237us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.500590 (+ 446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:22.566377 22143 ts_tablet_manager.cc:1428] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20250112 02:57:22.567338 22016 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 108) took 1116 ms. Trace:
I20250112 02:57:22.567739 22016 rpcz_store.cc:276] 0112 02:57:21.451044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.451176 (+ 132us) service_pool.cc:225] Handling call
0112 02:57:21.622562 (+171386us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.622684 (+ 122us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.766209 (+143525us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.563191 (+796982us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.567273 (+ 4082us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2044}
I20250112 02:57:22.571357 22143 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:22.573426 22561 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:22.587841 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 1e7d4e98b6fa4aecb5955e8603eac779 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:22.591114 21836 leader_election.cc:304] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:22.592010 22524 raft_consensus.cc:2798] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:22.592320 22524 raft_consensus.cc:491] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:22.592599 22524 raft_consensus.cc:3054] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:22.608069 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 0df734ba87264abd803237a5bbb32da2 (DEFAULT_TABLE table=default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.610110 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 67260685978247898ad224128ed8964f (DEFAULT_TABLE table=default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.611394 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0df734ba87264abd803237a5bbb32da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.612982 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67260685978247898ad224128ed8964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.615525 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb (DEFAULT_TABLE table=default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.616468 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.620430 22375 ts_tablet_manager.cc:1918] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:22.622967 22375 log.cc:1198] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:22.624202 22375 ts_tablet_manager.cc:1939] T 33a5c293bbd643aab9409c6e9abe4f0a P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:22.634714 22376 ts_tablet_manager.cc:1918] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:22.635298 22376 log.cc:1198] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/afeff74abcc643a5ad6565d9dcc40d62
I20250112 02:57:22.636314 22376 ts_tablet_manager.cc:1939] T afeff74abcc643a5ad6565d9dcc40d62 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:22.643256 22014 consensus_meta.cc:287] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.809s user 0.011s sys 0.000s
W20250112 02:57:22.643647 22018 consensus_meta.cc:287] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.793s user 0.008s sys 0.000s
W20250112 02:57:22.644003 22017 consensus_meta.cc:287] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.809s user 0.004s sys 0.003s
I20250112 02:57:22.647055 22020 tablet_service.cc:1467] Processing CreateTablet for tablet e21a0e4b5ebc426383cf332f46492b80 (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:22.655045 22015 consensus_meta.cc:287] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.826s user 0.005s sys 0.004s
I20250112 02:57:22.655546 22042 raft_consensus.cc:2463] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
W20250112 02:57:22.656462 22012 consensus_meta.cc:287] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.631s user 0.009s sys 0.000s
W20250112 02:57:22.656879 22019 consensus_meta.cc:287] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.847s user 0.009s sys 0.000s
I20250112 02:57:22.668664 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f7700b8cce4261a32f2009d1481b99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:22.588879 21691 leader_election.cc:343] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:22.596551 22379 ts_tablet_manager.cc:1918] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:22.670230 22379 log.cc:1198] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:22.670182 21691 leader_election.cc:304] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:22.671326 22379 ts_tablet_manager.cc:1939] T 12f556cd7d6f4e13a525d3b8d4207d94 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:22.672286 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 90) took 2664 ms. Trace:
I20250112 02:57:22.673179 22379 rpcz_store.cc:276] 0112 02:57:20.007936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.227699 (+219763us) service_pool.cc:225] Handling call
0112 02:57:21.866440 (+1638741us) ts_tablet_manager.cc:1145] Deleting tablet 12f556cd7d6f4e13a525d3b8d4207d94
0112 02:57:21.866492 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:22.263392 (+396900us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.596349 (+332957us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.672211 (+ 75862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638112}
I20250112 02:57:22.675112 22379 tablet_replica.cc:331] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:22.676784 22379 raft_consensus.cc:2238] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:57:22.671331 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 82544.9us
I20250112 02:57:22.678400 22006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8f7301bed6c4d98a80c12b70073e62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.678464 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_02f0da840c0646d59412dcb267a4d571 [id=ed996933e39d472992d6f351ae51fb39]) successfully deleted
I20250112 02:57:22.678794 22528 raft_consensus.cc:2743] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:22.696462 22379 raft_consensus.cc:2267] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:22.707398 21385 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_dc2380b6e80d458d8260d4bc07ded652" } new_table_name: "default.Table_6a8f570367d44d06b62d0d335dee914e" modify_external_catalogs: true
I20250112 02:57:22.710909 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e21a0e4b5ebc426383cf332f46492b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.711473 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_309cdb1f2ef042d0a115d9709cf410ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:22.713152 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_309cdb1f2ef042d0a115d9709cf410ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:22.734066 21389 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5] alter complete (version 1)
I20250112 02:57:22.740255 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 7d699356dd3341338670b433a7fa02ad
I20250112 02:57:22.748157 22013 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 117) took 1265 ms. Trace:
I20250112 02:57:22.751343 21724 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:22.753719 22379 ts_tablet_manager.cc:1905] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:22.750207 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:22.755178 22013 rpcz_store.cc:276] 0112 02:57:21.482795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.482953 (+ 158us) service_pool.cc:225] Handling call
0112 02:57:21.717822 (+234869us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.718728 (+ 906us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.851992 (+133264us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.566835 (+714843us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.748058 (+181223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1712}
I20250112 02:57:22.808743 22014 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 116) took 1327 ms. Trace:
I20250112 02:57:22.809283 22014 rpcz_store.cc:276] 0112 02:57:21.481214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.481351 (+ 137us) service_pool.cc:225] Handling call
0112 02:57:21.709803 (+228452us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.712998 (+ 3195us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.833670 (+120672us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.808089 (+974419us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.808659 (+ 570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164219}
I20250112 02:57:22.811283 22018 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 106) took 1365 ms. Trace:
I20250112 02:57:22.811627 22018 rpcz_store.cc:276] 0112 02:57:21.446333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.447808 (+ 1475us) service_pool.cc:225] Handling call
0112 02:57:21.595681 (+147873us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.595772 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.850452 (+254680us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.810791 (+960339us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.811225 (+ 434us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":166640}
I20250112 02:57:22.813571 22017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 111) took 1353 ms. Trace:
I20250112 02:57:22.814734 22015 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 115) took 1335 ms. Trace:
I20250112 02:57:22.815183 22015 rpcz_store.cc:276] 0112 02:57:21.479760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.479890 (+ 130us) service_pool.cc:225] Handling call
0112 02:57:21.706660 (+226770us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.709776 (+ 3116us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.828953 (+119177us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.814174 (+985221us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.814669 (+ 495us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":158560}
I20250112 02:57:22.818233 22019 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:53076 (request call id 107) took 1368 ms. Trace:
I20250112 02:57:22.818722 22019 rpcz_store.cc:276] 0112 02:57:21.449432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.449588 (+ 156us) service_pool.cc:225] Handling call
0112 02:57:21.618561 (+168973us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.618657 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.809812 (+191155us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.817677 (+1007865us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.818164 (+ 487us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":160286}
I20250112 02:57:22.813970 22017 rpcz_store.cc:276] 0112 02:57:21.459958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.460086 (+ 128us) service_pool.cc:225] Handling call
0112 02:57:21.634546 (+174460us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.634636 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:21.834195 (+199559us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.813037 (+978842us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:22.813480 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":168573}
I20250112 02:57:22.829941 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 76) took 3205 ms. Trace:
I20250112 02:57:22.830298 22376 rpcz_store.cc:276] 0112 02:57:19.624519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.624654 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:22.485103 (+2860449us) ts_tablet_manager.cc:1145] Deleting tablet afeff74abcc643a5ad6565d9dcc40d62
0112 02:57:22.485155 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:22.504439 (+ 19284us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.634312 (+129873us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:22.829886 (+195574us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":171026,"tablet-delete.queue_time_us":2859950}
I20250112 02:57:22.831470 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet afeff74abcc643a5ad6565d9dcc40d62 (table default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]) successfully deleted
I20250112 02:57:22.834276 22375 tablet_replica.cc:331] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:22.835016 22375 raft_consensus.cc:2238] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:22.835493 22375 raft_consensus.cc:2267] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:22.837083 22375 ts_tablet_manager.cc:1905] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:22.759357 21755 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.021s sys 0.036s
W20250112 02:57:22.840713 21755 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.624s user 0.021s sys 0.036s
I20250112 02:57:22.791172 22528 raft_consensus.cc:491] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:22.841602 22528 raft_consensus.cc:513] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.844034 22528 leader_election.cc:290] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:22.845450 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:22.846114 22042 raft_consensus.cc:2388] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:22.851017 22566 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:22.851562 22566 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:22.854002 22566 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:22.855161 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 196 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:22.801434 22560 tablet.cc:1721] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:22.842787 22144 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.162s sys 0.261s
W20250112 02:57:22.866823 22144 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.670s user 0.162s sys 0.262s
I20250112 02:57:22.867502 21882 tablet_service.cc:1467] Processing CreateTablet for tablet d6ca275360ec41ceb0f7ef9bfaedf647 (DEFAULT_TABLE table=default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.826491 22524 raft_consensus.cc:513] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.869984 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e81199f79d46d6922eb1c6805d25da" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:22.872706 21902 raft_consensus.cc:2388] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:22.873669 21691 leader_election.cc:304] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:22.874379 22528 raft_consensus.cc:3054] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:22.882061 21533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20250112 02:57:22.883105 22145 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:22.883507 22145 ts_tablet_manager.cc:1397] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.667s user 0.007s sys 0.005s
W20250112 02:57:22.889735 21533 leader_election.cc:336] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:22.890242 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:22.891145 22144 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:22.891537 22144 ts_tablet_manager.cc:1397] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.702s user 0.166s sys 0.263s
I20250112 02:57:22.904443 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6ca275360ec41ceb0f7ef9bfaedf647. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:22.908777 22145 raft_consensus.cc:357] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.909420 22145 raft_consensus.cc:383] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.909669 22145 raft_consensus.cc:738] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.910189 22145 consensus_queue.cc:260] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.912701 22566 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:22.915287 22414 raft_consensus.cc:491] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:22.915702 22414 raft_consensus.cc:513] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.917613 22414 leader_election.cc:290] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:22.924364 22528 raft_consensus.cc:2743] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:22.926723 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:22.927271 22042 raft_consensus.cc:2463] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:22.928755 21690 leader_election.cc:304] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:22.936422 22528 raft_consensus.cc:2798] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:22.936823 22528 raft_consensus.cc:491] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:22.937187 22528 raft_consensus.cc:3054] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:22.941673 22145 ts_tablet_manager.cc:1428] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20250112 02:57:22.966614 22145 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:22.972386 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 77) took 3330 ms. Trace:
I20250112 02:57:22.972695 22376 rpcz_store.cc:276] 0112 02:57:19.642242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.642364 (+ 122us) service_pool.cc:225] Handling call
0112 02:57:22.972047 (+3329683us) ts_tablet_manager.cc:1145] Deleting tablet afeff74abcc643a5ad6565d9dcc40d62
0112 02:57:22.972120 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:22.972334 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3188292}
I20250112 02:57:22.973634 22376 tablet_replica.cc:331] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:22.974879 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because the tablet was not found. No further retry: Not found: Tablet not found: afeff74abcc643a5ad6565d9dcc40d62
W20250112 02:57:22.976639 21980 consensus_peers.cc:487] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:22.977525 22145 tablet_bootstrap.cc:654] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.974257 22376 raft_consensus.cc:2238] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:22.979094 22376 raft_consensus.cc:2267] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:22.980118 22144 raft_consensus.cc:357] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.980799 22144 raft_consensus.cc:383] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:22.981055 22144 raft_consensus.cc:738] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:22.981739 22144 consensus_queue.cc:260] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.992173 22144 ts_tablet_manager.cc:1428] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.100s user 0.004s sys 0.002s
I20250112 02:57:22.992955 22144 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:22.994117 22457 raft_consensus.cc:491] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:22.994627 22457 raft_consensus.cc:513] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.996485 21387 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 161) took 1364 ms. Trace:
I20250112 02:57:22.996948 22457 leader_election.cc:290] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:22.998560 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:22.998914 21390 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 6587208a44b343509f387472c08c73c3
I20250112 02:57:22.999039 22144 tablet_bootstrap.cc:654] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:22.999473 21390 master_service.cc:946] ReplaceTablet: received request to replace tablet 301c8a2c534a4c54bfd6e678f3184ea4 from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:22.999894 21980 leader_election.cc:343] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.001258 21873 tablet_service.cc:1467] Processing CreateTablet for tablet e8f7301bed6c4d98a80c12b70073e62f (DEFAULT_TABLE table=default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:22.996817 21387 rpcz_store.cc:276] 0112 02:57:21.632270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.889645 (+257375us) service_pool.cc:225] Handling call
0112 02:57:21.952153 (+ 62508us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:22.167846 (+215693us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:22.995927 (+828081us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:22.996183 (+ 256us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:22.996414 (+ 231us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:22.869178 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:22.870204 (+ 1026us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:22.870233 (+ 29us) write_op.cc:435] Acquired schema lock
0112 02:57:22.912527 (+ 42294us) tablet.cc:659] Decoding operations
0112 02:57:22.912798 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:22.912833 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:22.912854 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:22.913042 (+ 188us) tablet.cc:698] Row locks acquired
0112 02:57:22.913055 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:22.913161 (+ 106us) write_op.cc:270] Start()
0112 02:57:22.913238 (+ 77us) write_op.cc:276] Timestamp: P: 1736650642913126 usec, L: 0
0112 02:57:22.913252 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:22.913711 (+ 459us) log.cc:844] Serialized 433 byte log entry
0112 02:57:22.986341 (+ 72630us) op_driver.cc:464] REPLICATION: finished
0112 02:57:22.989278 (+ 2937us) write_op.cc:301] APPLY: starting
0112 02:57:22.989322 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:22.989422 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:22.989540 (+ 118us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:22.989908 (+ 368us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:22.990036 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:22.990077 (+ 41us) write_op.cc:312] APPLY: finished
0112 02:57:22.990214 (+ 137us) log.cc:844] Serialized 29 byte log entry
0112 02:57:22.990320 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:22.990368 (+ 48us) write_op.cc:454] Released schema lock
0112 02:57:22.990390 (+ 22us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2817,"apply.run_cpu_time_us":1289,"apply.run_wall_time_us":9352,"num_ops":2,"prepare.queue_time_us":628779,"prepare.run_cpu_time_us":1822,"prepare.run_wall_time_us":87382,"replication_time_us":73009}]]}
I20250112 02:57:23.003475 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 77d80c6a60bc46449c7c0656128100c8 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:23.004540 22457 raft_consensus.cc:491] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.004987 22457 raft_consensus.cc:513] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.005982 22145 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.007337 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:23.007477 22457 leader_election.cc:290] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.007936 22375 ts_tablet_manager.cc:1918] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:23.008301 22375 log.cc:1198] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:23.009056 22375 ts_tablet_manager.cc:1939] T 5f787ffe65e34e8f9852d149fff9f947 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
W20250112 02:57:23.009145 21980 leader_election.cc:343] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.006352 22145 ts_tablet_manager.cc:1397] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.006s
I20250112 02:57:23.010684 21874 tablet_service.cc:1467] Processing CreateTablet for tablet e21a0e4b5ebc426383cf332f46492b80 (DEFAULT_TABLE table=default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.011237 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:23.011745 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e21a0e4b5ebc426383cf332f46492b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.011960 21902 raft_consensus.cc:2388] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:23.011883 22145 raft_consensus.cc:357] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.012480 22145 raft_consensus.cc:383] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.012696 22145 raft_consensus.cc:738] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.013218 22145 consensus_queue.cc:260] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.018462 22145 ts_tablet_manager.cc:1428] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250112 02:57:23.018842 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.018740 22528 raft_consensus.cc:513] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.019907 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.020541 22528 leader_election.cc:290] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
W20250112 02:57:23.020957 21978 leader_election.cc:343] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.021366 21978 leader_election.cc:304] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.022169 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
W20250112 02:57:23.022491 21978 leader_election.cc:343] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.022917 21978 leader_election.cc:304] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.023404 22042 raft_consensus.cc:3054] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.023934 22584 raft_consensus.cc:2743] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:23.024446 22584 raft_consensus.cc:2743] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:23.024750 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:23.025431 21902 raft_consensus.cc:2388] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:23.027321 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:23.028009 21754 raft_consensus.cc:2388] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:23.032060 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8f7301bed6c4d98a80c12b70073e62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.052943 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 84f7700b8cce4261a32f2009d1481b99 (DEFAULT_TABLE table=default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.054121 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84f7700b8cce4261a32f2009d1481b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.060132 22457 raft_consensus.cc:491] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.060534 22457 raft_consensus.cc:513] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:22.965190 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 82d2ac91c5554a64936f364e4e1978f8 (table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab]). Expected version 1 got 0
I20250112 02:57:23.062446 22457 leader_election.cc:290] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.062726 22524 leader_election.cc:290] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.062847 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:23.064853 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:23.067268 21980 leader_election.cc:343] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.069401 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:23.070587 21978 leader_election.cc:343] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.070959 21978 leader_election.cc:304] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.071164 21736 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46754 (request call id 116) took 1057 ms. Trace:
I20250112 02:57:23.071935 22457 raft_consensus.cc:2743] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:23.071962 21736 rpcz_store.cc:276] 0112 02:57:22.014091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.014189 (+ 98us) service_pool.cc:225] Handling call
0112 02:57:22.097682 (+ 83493us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:22.097754 (+ 72us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:22.628716 (+530962us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.070774 (+442058us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:23.071116 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126207}
I20250112 02:57:23.073436 22143 tablet_bootstrap.cc:654] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.079384 21727 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46754 (request call id 115) took 1070 ms. Trace:
I20250112 02:57:23.079701 21727 rpcz_store.cc:276] 0112 02:57:22.008715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.008811 (+ 96us) service_pool.cc:225] Handling call
0112 02:57:22.010203 (+ 1392us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:22.010278 (+ 75us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:22.665153 (+654875us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.078941 (+413788us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:23.079334 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:23.081357 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:23.082289 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:23.087074 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:23.088012 21734 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:46754 (request call id 111) took 1200 ms. Trace:
I20250112 02:57:23.088460 21734 rpcz_store.cc:276] 0112 02:57:21.887999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.927651 (+ 39652us) service_pool.cc:225] Handling call
0112 02:57:21.953258 (+ 25607us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0112 02:57:21.953327 (+ 69us) ts_tablet_manager.cc:676] Creating new metadata...
0112 02:57:22.833422 (+880095us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.087436 (+254014us) tablet_replica.cc:206] Creating consensus instance
0112 02:57:23.087939 (+ 503us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:23.091496 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]) successfully deleted
I20250112 02:57:23.091933 22376 ts_tablet_manager.cc:1905] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.093524 22145 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.096647 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:23.098304 22145 tablet_bootstrap.cc:654] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.112438 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 110) took 1137 ms. Trace:
I20250112 02:57:23.112927 22375 rpcz_store.cc:276] 0112 02:57:21.975355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.975800 (+ 445us) service_pool.cc:225] Handling call
0112 02:57:22.834141 (+858341us) ts_tablet_manager.cc:1145] Deleting tablet 5f787ffe65e34e8f9852d149fff9f947
0112 02:57:22.834181 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:22.910661 (+ 76480us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.007671 (+ 97010us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.112389 (+104718us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5,"tablet-delete.queue_time_us":857787}
I20250112 02:57:23.114531 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 113) took 1029 ms. Trace:
I20250112 02:57:23.115124 22375 rpcz_store.cc:276] 0112 02:57:22.085224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.085359 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:23.114217 (+1028858us) ts_tablet_manager.cc:1145] Deleting tablet f9bb62ab3f274c29b2c5f9528890b816
0112 02:57:23.114266 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.114463 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028323}
I20250112 02:57:23.117499 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 114) took 1012 ms. Trace:
I20250112 02:57:23.117843 22375 rpcz_store.cc:276] 0112 02:57:22.105353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.105485 (+ 132us) service_pool.cc:225] Handling call
0112 02:57:23.117266 (+1011781us) ts_tablet_manager.cc:1145] Deleting tablet 5890ebb3c55943d5a334d6c93fbdaf0e
0112 02:57:23.117298 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.117455 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1011219}
W20250112 02:57:23.142413 21379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250112 02:57:23.148813 21383 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3cc36619a0114526980225e5fb00e357" } new_table_name: "default.Table_c7d67cd6dc514abb8a8fef6106316336" modify_external_catalogs: true
I20250112 02:57:23.156642 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 115) took 1049 ms. Trace:
I20250112 02:57:23.156980 22375 rpcz_store.cc:276] 0112 02:57:22.106751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.106863 (+ 112us) service_pool.cc:225] Handling call
0112 02:57:23.156372 (+1049509us) ts_tablet_manager.cc:1145] Deleting tablet 12f556cd7d6f4e13a525d3b8d4207d94
0112 02:57:23.156421 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.156587 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049050}
I20250112 02:57:23.159380 22042 raft_consensus.cc:2463] T aab2cf8bc28b47239eaa3dcaa4c90e9c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:23.160441 22145 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.160439 21690 leader_election.cc:304] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.160873 22145 ts_tablet_manager.cc:1397] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.067s user 0.012s sys 0.000s
I20250112 02:57:23.161720 22528 raft_consensus.cc:2798] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:23.163542 22528 raft_consensus.cc:695] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:23.164283 22145 raft_consensus.cc:357] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.164883 22145 raft_consensus.cc:383] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.165077 22145 raft_consensus.cc:738] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.164306 22528 consensus_queue.cc:237] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.167204 22145 consensus_queue.cc:260] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.181967 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 116) took 1073 ms. Trace:
I20250112 02:57:23.182340 22375 rpcz_store.cc:276] 0112 02:57:22.108282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.108388 (+ 106us) service_pool.cc:225] Handling call
0112 02:57:23.158450 (+1050062us) ts_tablet_manager.cc:1145] Deleting tablet 3781431324bc479d97bd22b6bb1243b9
0112 02:57:23.158481 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.181916 (+ 23435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049539}
I20250112 02:57:23.183495 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 117) took 1073 ms. Trace:
I20250112 02:57:23.183857 22375 rpcz_store.cc:276] 0112 02:57:22.109811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.109917 (+ 106us) service_pool.cc:225] Handling call
0112 02:57:23.183255 (+1073338us) ts_tablet_manager.cc:1145] Deleting tablet 35b39d0cf44f4e91908b8e568df0a16a
0112 02:57:23.183293 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.183448 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072831}
I20250112 02:57:23.197504 22145 ts_tablet_manager.cc:1428] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.036s user 0.005s sys 0.000s
I20250112 02:57:23.205273 22145 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.219959 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a1ba8d2b677e47f5bffefd5fa58046e4" } modify_external_catalogs: true
I20250112 02:57:23.220455 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_a1ba8d2b677e47f5bffefd5fa58046e4" } modify_external_catalogs: true
I20250112 02:57:23.221691 22561 tablet.cc:1721] T 82d2ac91c5554a64936f364e4e1978f8 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:23.227921 22143 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:23.228315 22143 ts_tablet_manager.cc:1397] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.657s user 0.016s sys 0.015s
I20250112 02:57:23.229238 22145 tablet_bootstrap.cc:654] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.230453 22143 raft_consensus.cc:357] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.231134 22143 raft_consensus.cc:383] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.231384 22143 raft_consensus.cc:738] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.188309 22144 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:23.235265 22144 ts_tablet_manager.cc:1397] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.242s user 0.009s sys 0.000s
I20250112 02:57:23.235216 22143 consensus_queue.cc:260] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.237567 22144 raft_consensus.cc:357] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.238248 22144 raft_consensus.cc:383] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.238562 22144 raft_consensus.cc:738] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.239181 22144 consensus_queue.cc:260] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.251582 22145 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.251953 22145 ts_tablet_manager.cc:1397] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.000s
W20250112 02:57:23.253062 21690 consensus_peers.cc:487] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afeff74abcc643a5ad6565d9dcc40d62. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:23.253965 22145 raft_consensus.cc:357] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.254648 22145 raft_consensus.cc:383] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.254871 22145 raft_consensus.cc:738] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.256758 22143 ts_tablet_manager.cc:1428] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20250112 02:57:23.259284 22145 consensus_queue.cc:260] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.263602 22144 ts_tablet_manager.cc:1428] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20250112 02:57:23.112429 21836 leader_election.cc:304] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
W20250112 02:57:23.264870 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 152599us
I20250112 02:57:23.265619 21390 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:23.266840 21390 catalog_manager.cc:6596] ReplaceTablet: tablet 301c8a2c534a4c54bfd6e678f3184ea4 deleted and replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4
I20250112 02:57:23.269039 21836 leader_election.cc:304] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:23.243081 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 118) took 1132 ms. Trace:
I20250112 02:57:23.269707 22375 rpcz_store.cc:276] 0112 02:57:22.111130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.111247 (+ 117us) service_pool.cc:225] Handling call
0112 02:57:23.242795 (+1131548us) ts_tablet_manager.cc:1145] Deleting tablet 402c24e54ab040e29949b1e4efbe5ff3
0112 02:57:23.242845 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.243023 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1131023}
I20250112 02:57:23.270670 22375 tablet_replica.cc:331] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:23.271281 21731 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:23.271374 22375 raft_consensus.cc:2238] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:23.226135 22603 tablet.cc:1721] T 82d2ac91c5554a64936f364e4e1978f8 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:23.207922 22524 raft_consensus.cc:491] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.272059 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:23.272011 22528 consensus_queue.cc:1035] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:23.271831 22524 raft_consensus.cc:513] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.274719 22524 leader_election.cc:290] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.265431 22587 raft_consensus.cc:2798] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:23.276038 22585 raft_consensus.cc:2743] T aab2cf8bc28b47239eaa3dcaa4c90e9c P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:57:23.277384 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.278110 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 5f787ffe65e34e8f9852d149fff9f947 (table default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea [id=e084f29dae5a41899d7cb805c1ea63b6]) successfully deleted
W20250112 02:57:23.278705 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet f9bb62ab3f274c29b2c5f9528890b816 because the tablet was not found. No further retry: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816
W20250112 02:57:23.279206 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5890ebb3c55943d5a334d6c93fbdaf0e because the tablet was not found. No further retry: Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e
W20250112 02:57:23.279717 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 12f556cd7d6f4e13a525d3b8d4207d94 because the tablet was not found. No further retry: Not found: Tablet not found: 12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:23.280812 22145 ts_tablet_manager.cc:1428] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250112 02:57:23.283695 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_df360ac1d7fb44eb8e31fd606b361d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:23.285267 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df360ac1d7fb44eb8e31fd606b361d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:23.293340 22145 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.298368 22144 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:23.301774 21383 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a8273b629804f1c86a813b04747a1aa consensus state has no leader
W20250112 02:57:23.304540 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 3781431324bc479d97bd22b6bb1243b9 because the tablet was not found. No further retry: Not found: Tablet not found: 3781431324bc479d97bd22b6bb1243b9
W20250112 02:57:23.306190 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 35b39d0cf44f4e91908b8e568df0a16a because the tablet was not found. No further retry: Not found: Tablet not found: 35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:23.311298 21383 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed" } modify_external_catalogs: true
I20250112 02:57:23.311837 21383 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed" } modify_external_catalogs: true
W20250112 02:57:23.313201 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_c9690c1ccf2845e4a070fb8e6069cd60 [id=55d68af4c9444c96828bd8d85c3bdaf8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
I20250112 02:57:23.318746 22145 tablet_bootstrap.cc:654] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.318955 22376 ts_tablet_manager.cc:1918] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:23.330689 22376 log.cc:1198] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/3781431324bc479d97bd22b6bb1243b9
I20250112 02:57:23.331408 22376 ts_tablet_manager.cc:1939] T 3781431324bc479d97bd22b6bb1243b9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:23.336172 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_be69112a66094ce6871bfe8e390e087f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:23.337680 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be69112a66094ce6871bfe8e390e087f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:23.348008 21835 leader_election.cc:343] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.348685 22145 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.349116 22145 ts_tablet_manager.cc:1397] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.007s
I20250112 02:57:23.351150 22143 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:23.351246 22145 raft_consensus.cc:357] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.352031 22145 raft_consensus.cc:383] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.352423 22145 raft_consensus.cc:738] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.353554 22145 consensus_queue.cc:260] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.359730 22587 raft_consensus.cc:695] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:23.360460 22587 consensus_queue.cc:237] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.365228 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e2cfbf3f568b436f8d819d76e84c68e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:23.366686 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2cfbf3f568b436f8d819d76e84c68e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:23.365046 22143 tablet_bootstrap.cc:654] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.383592 22375 raft_consensus.cc:2267] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.385466 22375 ts_tablet_manager.cc:1905] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.392474 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:23.393052 22044 raft_consensus.cc:2463] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:23.394734 21876 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:23.397091 21836 leader_election.cc:304] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:23.399070 22145 ts_tablet_manager.cc:1428] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.050s user 0.002s sys 0.003s
I20250112 02:57:23.402230 22145 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.403158 22611 tablet.cc:1721] T 82d2ac91c5554a64936f364e4e1978f8 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:23.391762 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 402c24e54ab040e29949b1e4efbe5ff3 because the tablet was not found. No further retry: Not found: Tablet not found: 402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:23.417912 22145 tablet_bootstrap.cc:654] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.421725 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 81) took 3744 ms. Trace:
I20250112 02:57:23.422032 22376 rpcz_store.cc:276] 0112 02:57:19.677731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:19.677846 (+ 115us) service_pool.cc:225] Handling call
0112 02:57:22.973499 (+3295653us) ts_tablet_manager.cc:1145] Deleting tablet 3781431324bc479d97bd22b6bb1243b9
0112 02:57:22.973539 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.213766 (+240227us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.318735 (+104969us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.421668 (+102933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3286575}
I20250112 02:57:23.422948 22376 tablet_replica.cc:331] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:23.422932 22379 ts_tablet_manager.cc:1918] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:23.423259 22379 log.cc:1198] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/bb324cd3120e45449e0500c99338135d
I20250112 02:57:23.423543 22376 raft_consensus.cc:2238] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:23.423887 22379 ts_tablet_manager.cc:1939] T bb324cd3120e45449e0500c99338135d P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:23.423969 22376 raft_consensus.cc:2267] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.425443 22376 ts_tablet_manager.cc:1905] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.427052 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 3781431324bc479d97bd22b6bb1243b9 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:23.428364 22568 raft_consensus.cc:491] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.428750 22568 raft_consensus.cc:513] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.430380 22568 leader_election.cc:290] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.432092 22143 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:23.432452 22143 ts_tablet_manager.cc:1397] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.081s user 0.004s sys 0.007s
I20250112 02:57:23.433305 22524 raft_consensus.cc:2798] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:23.433614 22524 raft_consensus.cc:491] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:23.433915 22524 raft_consensus.cc:3054] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.434463 22143 raft_consensus.cc:357] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.435645 22143 raft_consensus.cc:383] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.435861 22143 raft_consensus.cc:738] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.434821 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47e14720a704a19ac3aea8723e13c28" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:23.437281 21978 leader_election.cc:343] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.438884 22579 consensus_queue.cc:1035] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:23.440246 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47e14720a704a19ac3aea8723e13c28" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:23.440748 21902 raft_consensus.cc:2463] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:23.442342 21980 leader_election.cc:304] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
W20250112 02:57:23.442759 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 45760.2us
I20250112 02:57:23.442763 22143 consensus_queue.cc:260] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.443226 22568 raft_consensus.cc:2798] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:23.443497 22568 raft_consensus.cc:491] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:23.443770 22568 raft_consensus.cc:3054] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.444072 22457 raft_consensus.cc:491] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.444461 22457 raft_consensus.cc:513] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.446166 22457 leader_election.cc:290] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.447153 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:23.449226 21390 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:23.450583 21980 leader_election.cc:343] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.460068 22579 consensus_queue.cc:1035] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:23.448210 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.462682 21753 raft_consensus.cc:2463] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:23.463541 21978 leader_election.cc:304] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.464959 22144 tablet_bootstrap.cc:654] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.467875 22457 raft_consensus.cc:2798] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:23.468223 22457 raft_consensus.cc:491] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:23.468564 22457 raft_consensus.cc:3054] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.469444 22143 ts_tablet_manager.cc:1428] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.037s user 0.005s sys 0.002s
I20250112 02:57:23.470250 22143 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:23.483096 22579 raft_consensus.cc:491] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.483539 22579 raft_consensus.cc:513] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.485316 22579 leader_election.cc:290] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.486937 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:23.487030 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.487533 22044 raft_consensus.cc:2463] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
W20250112 02:57:23.488812 21835 leader_election.cc:343] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.489888 22143 tablet_bootstrap.cc:654] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.494558 21836 leader_election.cc:304] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
W20250112 02:57:23.496901 21390 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:23.504946 22579 raft_consensus.cc:2798] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:23.505270 22579 raft_consensus.cc:491] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:23.505544 22579 raft_consensus.cc:3054] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.521396 22524 raft_consensus.cc:513] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.521831 22568 raft_consensus.cc:513] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.523345 22524 leader_election.cc:290] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.528113 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:23.528640 22044 raft_consensus.cc:3054] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.528720 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47e14720a704a19ac3aea8723e13c28" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:23.529224 21900 raft_consensus.cc:3054] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.535759 22568 leader_election.cc:290] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.545501 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 92) took 3295 ms. Trace:
I20250112 02:57:23.556737 21390 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_c9690c1ccf2845e4a070fb8e6069cd60" } new_table_name: "default.Table_142892c5113b4defbea191ff5b1021cf" modify_external_catalogs: true
I20250112 02:57:23.561758 22145 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.561839 22144 tablet_bootstrap.cc:492] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:23.562137 22145 ts_tablet_manager.cc:1397] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.160s user 0.007s sys 0.000s
I20250112 02:57:23.562227 22144 ts_tablet_manager.cc:1397] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.264s user 0.009s sys 0.000s
I20250112 02:57:23.564314 22144 raft_consensus.cc:357] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.564944 22144 raft_consensus.cc:383] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.565155 22144 raft_consensus.cc:738] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.565711 22144 consensus_queue.cc:260] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.573100 22145 raft_consensus.cc:357] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.574808 22145 raft_consensus.cc:383] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.575052 22145 raft_consensus.cc:738] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.583029 22145 consensus_queue.cc:260] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.587430 22145 ts_tablet_manager.cc:1428] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250112 02:57:23.592881 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 (DEFAULT_TABLE table=default.Table_582c4ae70bd94161908109db890c099e [id=097357eb9ce548aa9fe2db1a906c70d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.594839 22145 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.545831 22379 rpcz_store.cc:276] 0112 02:57:20.249822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.249946 (+ 124us) service_pool.cc:225] Handling call
0112 02:57:22.674912 (+2424966us) ts_tablet_manager.cc:1145] Deleting tablet bb324cd3120e45449e0500c99338135d
0112 02:57:22.674967 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.316270 (+641303us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.422625 (+106355us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.545441 (+122816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17620,"tablet-delete.queue_time_us":2424158}
I20250112 02:57:23.596612 22379 tablet_replica.cc:331] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:23.597563 22379 raft_consensus.cc:2238] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:23.600188 22379 raft_consensus.cc:2267] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.601804 22379 ts_tablet_manager.cc:1905] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.607586 22145 tablet_bootstrap.cc:654] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.611441 22457 raft_consensus.cc:513] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.613199 22457 leader_election.cc:290] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.613801 21900 raft_consensus.cc:2463] T f47e14720a704a19ac3aea8723e13c28 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:23.618222 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2601daad8ba742ec88ccfe603440e8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.624064 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:23.627218 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 82825c2061d74cafa814912bec005654 (DEFAULT_TABLE table=default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec [id=f6f1bd33b18440f0b4d05583eb796633]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:23.627846 21835 leader_election.cc:343] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.628198 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82825c2061d74cafa814912bec005654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.629729 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet bb324cd3120e45449e0500c99338135d (table default.Table_b9cd8852182c4c8f8988154152c71bc7 [id=362e3eb08b9f4322ba6903fabfd61eca]) successfully deleted
I20250112 02:57:23.630180 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 368af02998984cb2bee487b15b6a2b35 (DEFAULT_TABLE table=default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.630373 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47e14720a704a19ac3aea8723e13c28" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:23.631182 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368af02998984cb2bee487b15b6a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.634635 21731 tablet_service.cc:1467] Processing CreateTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 (DEFAULT_TABLE table=default.Table_582c4ae70bd94161908109db890c099e [id=097357eb9ce548aa9fe2db1a906c70d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.635622 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2601daad8ba742ec88ccfe603440e8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.639349 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 82825c2061d74cafa814912bec005654 (DEFAULT_TABLE table=default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec [id=f6f1bd33b18440f0b4d05583eb796633]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.640652 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82825c2061d74cafa814912bec005654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.647396 21980 leader_election.cc:304] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:23.649312 22568 raft_consensus.cc:2798] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:23.649400 22020 tablet_service.cc:1467] Processing CreateTablet for tablet 82825c2061d74cafa814912bec005654 (DEFAULT_TABLE table=default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec [id=f6f1bd33b18440f0b4d05583eb796633]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.649756 22568 raft_consensus.cc:695] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:23.650746 22568 consensus_queue.cc:237] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.653415 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 (DEFAULT_TABLE table=default.Table_582c4ae70bd94161908109db890c099e [id=097357eb9ce548aa9fe2db1a906c70d8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:23.655128 21978 leader_election.cc:343] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.657742 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:23.658520 21755 raft_consensus.cc:3054] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.665958 22144 ts_tablet_manager.cc:1428] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20250112 02:57:23.666215 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36495fb158a440309eddedd3910a7ad3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:23.666755 21900 raft_consensus.cc:3054] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.677585 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 368af02998984cb2bee487b15b6a2b35 (DEFAULT_TABLE table=default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.679472 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 118ef6d7c16941598177b92a94c4b472 (DEFAULT_TABLE table=default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.679852 21727 tablet_service.cc:1467] Processing CreateTablet for tablet 118ef6d7c16941598177b92a94c4b472 (DEFAULT_TABLE table=default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.682936 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 368af02998984cb2bee487b15b6a2b35 (DEFAULT_TABLE table=default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.683610 21874 tablet_service.cc:1467] Processing CreateTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a (DEFAULT_TABLE table=default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.684326 22010 tablet_service.cc:1467] Processing CreateTablet for tablet 118ef6d7c16941598177b92a94c4b472 (DEFAULT_TABLE table=default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.685963 21736 tablet_service.cc:1467] Processing CreateTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a (DEFAULT_TABLE table=default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.686540 22023 tablet_service.cc:1467] Processing CreateTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a (DEFAULT_TABLE table=default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.701838 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 46f8052651fa496cbb3702c17f592106 (DEFAULT_TABLE table=default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.701114 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 46f8052651fa496cbb3702c17f592106 (DEFAULT_TABLE table=default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.709308 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f445a4bf43db4571b6d3a4afe8fe823a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.709352 22579 raft_consensus.cc:513] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.710714 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f8052651fa496cbb3702c17f592106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.711511 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368af02998984cb2bee487b15b6a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.713469 22376 ts_tablet_manager.cc:1918] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:23.713825 22376 log.cc:1198] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/11fbebff35c344bdb305ea1a6dbbefe4
I20250112 02:57:23.714547 22376 ts_tablet_manager.cc:1939] T 11fbebff35c344bdb305ea1a6dbbefe4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:23.714995 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118ef6d7c16941598177b92a94c4b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.715219 22020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82825c2061d74cafa814912bec005654. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.716408 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368af02998984cb2bee487b15b6a2b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.717636 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118ef6d7c16941598177b92a94c4b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.718739 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f445a4bf43db4571b6d3a4afe8fe823a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.718706 22379 ts_tablet_manager.cc:1918] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:23.719019 22379 log.cc:1198] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:23.719673 22379 ts_tablet_manager.cc:1939] T 5890ebb3c55943d5a334d6c93fbdaf0e P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:23.720028 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2601daad8ba742ec88ccfe603440e8e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.719925 21756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
W20250112 02:57:23.721110 21835 leader_election.cc:343] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.721483 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 93) took 3398 ms. Trace:
I20250112 02:57:23.721582 22579 leader_election.cc:290] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.721765 22379 rpcz_store.cc:276] 0112 02:57:20.323013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.325595 (+ 2582us) service_pool.cc:225] Handling call
0112 02:57:23.596424 (+3270829us) ts_tablet_manager.cc:1145] Deleting tablet 5890ebb3c55943d5a334d6c93fbdaf0e
0112 02:57:23.596489 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.612813 (+ 16324us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.711830 (+ 99017us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.721430 (+ 9600us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269844}
I20250112 02:57:23.727885 22006 tablet_service.cc:1467] Processing CreateTablet for tablet 46f8052651fa496cbb3702c17f592106 (DEFAULT_TABLE table=default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.728919 22006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f8052651fa496cbb3702c17f592106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.732679 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 94) took 3405 ms. Trace:
I20250112 02:57:23.732661 22457 raft_consensus.cc:491] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.733065 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 2722e4001ba44a7998f4ba7682bd3c44 (DEFAULT_TABLE table=default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.732987 22379 rpcz_store.cc:276] 0112 02:57:20.326972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.327344 (+ 372us) service_pool.cc:225] Handling call
0112 02:57:23.732453 (+3405109us) ts_tablet_manager.cc:1145] Deleting tablet cad224c5ad844512b50da5190cfd1861
0112 02:57:23.732489 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.732632 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3404620}
I20250112 02:57:23.734261 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2722e4001ba44a7998f4ba7682bd3c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.734707 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 95) took 3407 ms. Trace:
I20250112 02:57:23.734886 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 5890ebb3c55943d5a334d6c93fbdaf0e (table default.Table_e8c3cd631d684a6b9d725c23343be273 [id=ebd1895e6bf64b67a7f3089109038c7d]) successfully deleted
W20250112 02:57:23.735553 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet cad224c5ad844512b50da5190cfd1861 because the tablet was not found. No further retry: Not found: Tablet not found: cad224c5ad844512b50da5190cfd1861
I20250112 02:57:23.735790 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 2722e4001ba44a7998f4ba7682bd3c44 (DEFAULT_TABLE table=default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.733078 22457 raft_consensus.cc:513] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.748188 22457 leader_election.cc:290] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.740893 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 2722e4001ba44a7998f4ba7682bd3c44 (DEFAULT_TABLE table=default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:23.751395 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2722e4001ba44a7998f4ba7682bd3c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.752465 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:23.753791 21980 leader_election.cc:343] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.758718 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:23.759308 21753 raft_consensus.cc:2463] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:23.760406 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:23.760980 22042 raft_consensus.cc:3054] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.734974 22379 rpcz_store.cc:276] 0112 02:57:20.327263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.331726 (+ 4463us) service_pool.cc:225] Handling call
0112 02:57:23.734475 (+3402749us) ts_tablet_manager.cc:1145] Deleting tablet 27234f3e69d544bdb53fb887874c2c0a
0112 02:57:23.734528 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.734660 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3402212}
I20250112 02:57:23.762290 22379 tablet_replica.cc:331] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:23.763011 22379 raft_consensus.cc:2238] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:23.762861 21978 leader_election.cc:304] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.763432 22379 raft_consensus.cc:2267] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.763736 22457 raft_consensus.cc:2798] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:23.764087 22457 raft_consensus.cc:491] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:57:23.764130 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 27234f3e69d544bdb53fb887874c2c0a because the tablet was not found. No further retry: Not found: Tablet not found: 27234f3e69d544bdb53fb887874c2c0a
I20250112 02:57:23.764493 22457 raft_consensus.cc:3054] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:23.765023 22379 ts_tablet_manager.cc:1905] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.776640 22375 ts_tablet_manager.cc:1918] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:23.777020 22375 log.cc:1198] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/7d699356dd3341338670b433a7fa02ad
I20250112 02:57:23.777127 22145 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.777487 22145 ts_tablet_manager.cc:1397] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.183s user 0.018s sys 0.006s
I20250112 02:57:23.777833 22375 ts_tablet_manager.cc:1939] T 7d699356dd3341338670b433a7fa02ad P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:23.781507 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118ef6d7c16941598177b92a94c4b472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.781985 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2722e4001ba44a7998f4ba7682bd3c44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.782707 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46f8052651fa496cbb3702c17f592106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.783248 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f445a4bf43db4571b6d3a4afe8fe823a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:23.786610 22143 tablet_bootstrap.cc:492] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:23.786988 22143 ts_tablet_manager.cc:1397] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.317s user 0.016s sys 0.016s
I20250112 02:57:23.789160 22143 raft_consensus.cc:357] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.799062 22143 raft_consensus.cc:383] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.799325 22143 raft_consensus.cc:738] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.808883 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:23.810035 21387 catalog_manager.cc:6596] ReplaceTablet: tablet 77d80c6a60bc46449c7c0656128100c8 deleted and replaced by tablet 66fe67a814504a89b6fa8f46e2a4aee9
I20250112 02:57:23.811337 22044 raft_consensus.cc:2463] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:23.812247 21900 raft_consensus.cc:2463] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:23.812561 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d80c6a60bc46449c7c0656128100c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66fe67a814504a89b6fa8f46e2a4aee9) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:23.813210 21755 raft_consensus.cc:2463] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:23.813469 21980 leader_election.cc:304] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:23.815433 22143 consensus_queue.cc:260] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.816040 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 92) took 3744 ms. Trace:
I20250112 02:57:23.816730 22568 raft_consensus.cc:2798] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:23.816466 22376 rpcz_store.cc:276] 0112 02:57:20.071798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.071889 (+ 91us) service_pool.cc:225] Handling call
0112 02:57:23.422823 (+3350934us) ts_tablet_manager.cc:1145] Deleting tablet 11fbebff35c344bdb305ea1a6dbbefe4
0112 02:57:23.422864 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.536477 (+113613us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.713251 (+176774us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:23.815990 (+102739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3350191}
I20250112 02:57:23.817201 22568 raft_consensus.cc:695] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:23.818106 22376 tablet_replica.cc:331] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:23.817921 22568 consensus_queue.cc:237] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.818918 21836 leader_election.cc:304] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:23.820134 22376 raft_consensus.cc:2238] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:23.820637 22376 raft_consensus.cc:2267] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.821549 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 11fbebff35c344bdb305ea1a6dbbefe4 (table default.Table_04002b2285af49eb9d1fd9edeb48810c [id=1308d8a9ae7d485489daf139f579b34c]) successfully deleted
I20250112 02:57:23.822227 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d80c6a60bc46449c7c0656128100c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66fe67a814504a89b6fa8f46e2a4aee9) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:23.826314 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d80c6a60bc46449c7c0656128100c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66fe67a814504a89b6fa8f46e2a4aee9) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:23.827246 22143 ts_tablet_manager.cc:1428] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.040s user 0.004s sys 0.001s
I20250112 02:57:23.831341 22143 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:23.833186 22579 raft_consensus.cc:2798] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:23.835568 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:23.840960 21731 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:23.843174 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:23.814453 22145 raft_consensus.cc:357] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:23.822444 22584 raft_consensus.cc:491] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:23.844987 21386 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 82d2ac91c5554a64936f364e4e1978f8 (table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab]). Expected version 1 got 0
I20250112 02:57:23.844761 22584 raft_consensus.cc:513] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.846220 21386 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:23.840183 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:23.846534 21980 consensus_peers.cc:487] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:23.846727 22376 ts_tablet_manager.cc:1905] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.846925 22584 leader_election.cc:290] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:23.847044 21386 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:23.847663 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:23.847913 22145 raft_consensus.cc:383] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.848167 22145 raft_consensus.cc:738] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.848129 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:23.848764 21980 leader_election.cc:343] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:23.849242 21978 leader_election.cc:343] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:23.849558 21978 leader_election.cc:304] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:23.850348 22584 raft_consensus.cc:2743] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:23.848783 22145 consensus_queue.cc:260] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:23.834623 22579 raft_consensus.cc:695] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:23.855165 22579 consensus_queue.cc:237] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.856681 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 33f89398c7aa4f2a938a476db897c833 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]). Expected version 1 got 0
I20250112 02:57:23.857167 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 82d2ac91c5554a64936f364e4e1978f8 (table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab]). Expected version 1 got 0
I20250112 02:57:23.857796 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:23.863852 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]) successfully deleted
I20250112 02:57:23.864742 22375 tablet_replica.cc:331] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:23.865455 22375 raft_consensus.cc:2238] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:23.877758 22145 ts_tablet_manager.cc:1428] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.100s user 0.007s sys 0.000s
I20250112 02:57:23.881137 22143 tablet_bootstrap.cc:654] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.858373 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:23.887085 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_dc2380b6e80d458d8260d4bc07ded652 [id=682416c8b9f6464789d238eb4e2a1c96]). Expected version 2 got 1
I20250112 02:57:23.891966 22145 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.895920 22375 raft_consensus.cc:2267] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:23.896654 22144 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:23.913511 22145 tablet_bootstrap.cc:654] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.915191 22375 ts_tablet_manager.cc:1905] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:23.931581 22144 tablet_bootstrap.cc:654] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:23.960196 22145 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:23.960872 22145 ts_tablet_manager.cc:1397] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.000s
I20250112 02:57:23.963196 22145 raft_consensus.cc:357] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.965762 22145 raft_consensus.cc:383] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.968215 22143 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:23.968495 22145 raft_consensus.cc:738] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.968784 22143 ts_tablet_manager.cc:1397] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.138s user 0.002s sys 0.010s
I20250112 02:57:23.969221 22145 consensus_queue.cc:260] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.971758 22144 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:23.971103 22143 raft_consensus.cc:357] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:23.972227 22144 ts_tablet_manager.cc:1397] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.000s
I20250112 02:57:23.972438 22143 raft_consensus.cc:383] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:23.972672 22143 raft_consensus.cc:738] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:23.979854 22145 ts_tablet_manager.cc:1428] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250112 02:57:23.981040 22144 raft_consensus.cc:357] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:23.974735 22143 consensus_queue.cc:260] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:23.993479 22143 ts_tablet_manager.cc:1428] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250112 02:57:23.997679 22145 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:23.999878 22143 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:23.997975 22144 raft_consensus.cc:383] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.003191 22144 raft_consensus.cc:738] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.003878 22144 consensus_queue.cc:260] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.005504 22379 ts_tablet_manager.cc:1918] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:24.005807 22379 log.cc:1198] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:24.006419 22379 ts_tablet_manager.cc:1939] T 402c24e54ab040e29949b1e4efbe5ff3 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:24.008906 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 96) took 3676 ms. Trace:
I20250112 02:57:24.009202 22379 rpcz_store.cc:276] 0112 02:57:20.332613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.332699 (+ 86us) service_pool.cc:225] Handling call
0112 02:57:23.762161 (+3429462us) ts_tablet_manager.cc:1145] Deleting tablet 402c24e54ab040e29949b1e4efbe5ff3
0112 02:57:23.762199 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:23.866917 (+104718us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.005316 (+138399us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.008856 (+ 3540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428990}
I20250112 02:57:24.009312 22457 raft_consensus.cc:513] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.010174 22379 tablet_replica.cc:331] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:24.010929 22379 raft_consensus.cc:2238] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:24.011354 22379 raft_consensus.cc:2267] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.013286 22379 ts_tablet_manager.cc:1905] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.018759 22144 ts_tablet_manager.cc:1428] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.046s user 0.003s sys 0.003s
I20250112 02:57:24.022894 22145 tablet_bootstrap.cc:654] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.024587 22143 tablet_bootstrap.cc:654] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.025488 22144 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:24.035876 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:24.041258 22144 tablet_bootstrap.cc:654] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:24.037148 21980 leader_election.cc:343] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:24.044925 22414 raft_consensus.cc:491] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.045336 22414 raft_consensus.cc:513] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.045722 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:24.046262 21755 raft_consensus.cc:3054] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.047037 22414 leader_election.cc:290] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:24.048823 22143 tablet_bootstrap.cc:492] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:24.049226 22143 ts_tablet_manager.cc:1397] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.002s
I20250112 02:57:24.050994 22457 leader_election.cc:290] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.051497 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:24.051365 22143 raft_consensus.cc:357] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.051977 22143 raft_consensus.cc:383] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.052198 22143 raft_consensus.cc:738] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:24.052822 21691 leader_election.cc:343] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:24.052739 22143 consensus_queue.cc:260] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.058288 22145 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:24.058840 22145 ts_tablet_manager.cc:1397] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.001s
I20250112 02:57:24.059087 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:24.059682 22044 raft_consensus.cc:2463] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:24.060740 21690 leader_election.cc:304] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:24.061512 22145 raft_consensus.cc:357] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.061822 22414 raft_consensus.cc:2798] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:24.062134 22145 raft_consensus.cc:383] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.062137 22414 raft_consensus.cc:491] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:24.062434 22414 raft_consensus.cc:3054] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.069000 22143 ts_tablet_manager.cc:1428] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250112 02:57:24.062393 22145 raft_consensus.cc:738] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.070976 22145 consensus_queue.cc:260] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.072726 22143 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:24.079272 22145 ts_tablet_manager.cc:1428] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.020s user 0.003s sys 0.002s
I20250112 02:57:24.088857 22143 tablet_bootstrap.cc:654] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.081372 22145 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:24.099057 22144 tablet_bootstrap.cc:492] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:24.099433 22144 ts_tablet_manager.cc:1397] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.006s
I20250112 02:57:24.101562 22144 raft_consensus.cc:357] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.102200 22144 raft_consensus.cc:383] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.103632 22579 raft_consensus.cc:491] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.104027 22579 raft_consensus.cc:513] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.105646 22579 leader_election.cc:290] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.106936 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:24.107523 21753 raft_consensus.cc:2463] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:24.108377 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:24.108772 21835 leader_election.cc:304] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:24.108955 22044 raft_consensus.cc:2463] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:24.108876 21734 tablet_service.cc:1467] Processing CreateTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 (DEFAULT_TABLE table=default.Table_e2cfbf3f568b436f8d819d76e84c68e8 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.109894 22585 raft_consensus.cc:2798] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:24.110168 22585 raft_consensus.cc:491] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:24.110450 22585 raft_consensus.cc:3054] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.110780 21731 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9bc28dc5942608fb7224f9f058e27 (DEFAULT_TABLE table=default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.102407 22144 raft_consensus.cc:738] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.111791 22006 tablet_service.cc:1467] Processing CreateTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 (DEFAULT_TABLE table=default.Table_e2cfbf3f568b436f8d819d76e84c68e8 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.112368 22144 consensus_queue.cc:260] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.113714 22375 ts_tablet_manager.cc:1918] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:24.114055 22375 log.cc:1198] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:24.114706 22375 ts_tablet_manager.cc:1939] T 301c8a2c534a4c54bfd6e678f3184ea4 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:24.117372 22585 raft_consensus.cc:513] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.118201 22042 raft_consensus.cc:2463] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:24.118557 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6efb19f0cc4a96b957ec34c27ab836 (DEFAULT_TABLE table=default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.119354 22375 tablet_replica.cc:331] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:24.119727 21836 leader_election.cc:304] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2
I20250112 02:57:24.120323 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce633c5b3b4cbea17c171564fa72d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.120169 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:24.120056 22375 raft_consensus.cc:2238] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:24.120697 21753 raft_consensus.cc:3054] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.121053 22579 raft_consensus.cc:2798] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:24.121131 22375 raft_consensus.cc:2267] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.121554 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a9bc28dc5942608fb7224f9f058e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.121512 22579 raft_consensus.cc:695] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:24.121768 22585 leader_election.cc:290] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.122248 22579 consensus_queue.cc:237] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.127995 22375 ts_tablet_manager.cc:1905] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.128192 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6efb19f0cc4a96b957ec34c27ab836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.134099 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4dddfea8e8b49189186930fce46ecc5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:24.134800 22042 raft_consensus.cc:3054] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.135939 22144 ts_tablet_manager.cc:1428] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250112 02:57:24.143260 21755 raft_consensus.cc:2463] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:24.144447 21978 leader_election.cc:304] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:24.145323 22457 raft_consensus.cc:2798] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:24.146765 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2b248224f9416faf4bd83a853ec918 (DEFAULT_TABLE table=default.Table_309cdb1f2ef042d0a115d9709cf410ae [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.147727 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b248224f9416faf4bd83a853ec918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.147997 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2b248224f9416faf4bd83a853ec918 (DEFAULT_TABLE table=default.Table_309cdb1f2ef042d0a115d9709cf410ae [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.150956 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6efb19f0cc4a96b957ec34c27ab836 (DEFAULT_TABLE table=default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.151949 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6efb19f0cc4a96b957ec34c27ab836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.151707 22585 consensus_queue.cc:1035] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:24.155985 21727 tablet_service.cc:1467] Processing CreateTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.156306 22457 raft_consensus.cc:695] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:24.156991 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4b543ed18b34cd58a1c588ed7d079c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.157035 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6efb19f0cc4a96b957ec34c27ab836 (DEFAULT_TABLE table=default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.157102 22017 tablet_service.cc:1467] Processing CreateTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.157119 22457 consensus_queue.cc:237] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.158330 21882 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9bc28dc5942608fb7224f9f058e27 (DEFAULT_TABLE table=default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.159263 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a9bc28dc5942608fb7224f9f058e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.166517 22145 tablet_bootstrap.cc:654] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.166878 22010 tablet_service.cc:1467] Processing CreateTablet for tablet e4a9bc28dc5942608fb7224f9f058e27 (DEFAULT_TABLE table=default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.169056 21872 tablet_service.cc:1467] Processing CreateTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 (DEFAULT_TABLE table=default.Table_e2cfbf3f568b436f8d819d76e84c68e8 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.169766 22144 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:24.170049 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce633c5b3b4cbea17c171564fa72d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.182787 22006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ce633c5b3b4cbea17c171564fa72d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.186959 22017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4b543ed18b34cd58a1c588ed7d079c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.188159 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b248224f9416faf4bd83a853ec918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.127542 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 301c8a2c534a4c54bfd6e678f3184ea4 (table default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]) successfully deleted
W20250112 02:57:24.188848 21378 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 61356.3us
I20250112 02:57:24.189075 22143 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:24.189450 22143 ts_tablet_manager.cc:1397] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.010s
I20250112 02:57:24.189534 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b6efb19f0cc4a96b957ec34c27ab836. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.192214 21877 tablet_service.cc:1467] Processing CreateTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.193800 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4b543ed18b34cd58a1c588ed7d079c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.195523 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 443aa544f6df4d50a78c9effd306e216 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.196547 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2b248224f9416faf4bd83a853ec918 (DEFAULT_TABLE table=default.Table_309cdb1f2ef042d0a115d9709cf410ae [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.126456 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_c9690c1ccf2845e4a070fb8e6069cd60 [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 2 got 0
I20250112 02:57:24.198570 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:24.205190 22568 raft_consensus.cc:491] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.208792 22144 tablet_bootstrap.cc:654] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.212610 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443aa544f6df4d50a78c9effd306e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.215246 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:24.191573 22143 raft_consensus.cc:357] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.216244 22143 raft_consensus.cc:383] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.216497 22143 raft_consensus.cc:738] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.216399 21753 raft_consensus.cc:2463] T b4dddfea8e8b49189186930fce46ecc5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:24.217134 22143 consensus_queue.cc:260] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.216030 22379 ts_tablet_manager.cc:1918] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:24.220067 22379 log.cc:1198] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/f9bb62ab3f274c29b2c5f9528890b816
I20250112 02:57:24.222651 22379 ts_tablet_manager.cc:1939] T f9bb62ab3f274c29b2c5f9528890b816 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:24.231150 21835 leader_election.cc:304] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:24.231662 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4a9bc28dc5942608fb7224f9f058e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.231930 22585 raft_consensus.cc:2798] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:24.236662 22143 ts_tablet_manager.cc:1428] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.047s user 0.004s sys 0.002s
I20250112 02:57:24.237612 22143 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:24.241232 21392 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_c9690c1ccf2845e4a070fb8e6069cd60 [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 2 got 0
I20250112 02:57:24.242344 21392 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:24.243247 21392 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:24.244311 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 443aa544f6df4d50a78c9effd306e216 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.245362 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443aa544f6df4d50a78c9effd306e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.245273 22019 tablet_service.cc:1467] Processing CreateTablet for tablet 443aa544f6df4d50a78c9effd306e216 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.246284 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 443aa544f6df4d50a78c9effd306e216. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.247531 21728 tablet_service.cc:1467] Processing CreateTablet for tablet a69123c4ab384e0380c62e52b9f3b42d (DEFAULT_TABLE table=default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0 [id=8dbf245ab5354db7a93b54749e978c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.249076 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69123c4ab384e0380c62e52b9f3b42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.250074 22012 tablet_service.cc:1467] Processing CreateTablet for tablet a69123c4ab384e0380c62e52b9f3b42d (DEFAULT_TABLE table=default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0 [id=8dbf245ab5354db7a93b54749e978c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.251101 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69123c4ab384e0380c62e52b9f3b42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.251950 21735 tablet_service.cc:1467] Processing CreateTablet for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.252931 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:24.253316 21390 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
I20250112 02:57:24.254757 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e2b248224f9416faf4bd83a853ec918. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:24.295941 22020 consensus_meta.cc:287] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.001s
I20250112 02:57:24.297144 22042 raft_consensus.cc:2463] T b4dddfea8e8b49189186930fce46ecc5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
W20250112 02:57:24.297910 22008 consensus_meta.cc:287] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.517s user 0.007s sys 0.001s
W20250112 02:57:24.299031 22011 consensus_meta.cc:287] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.527s user 0.004s sys 0.004s
I20250112 02:57:24.313398 22414 raft_consensus.cc:513] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.316164 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:24.317018 22043 raft_consensus.cc:3054] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.318990 22414 leader_election.cc:290] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:24.326243 22375 ts_tablet_manager.cc:1918] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:24.326639 22375 log.cc:1198] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:24.327308 22375 ts_tablet_manager.cc:1939] T 77d80c6a60bc46449c7c0656128100c8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:24.334445 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]). Expected version 1 got 0
I20250112 02:57:24.335125 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:24.335783 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 3 got 0
I20250112 02:57:24.205703 22568 raft_consensus.cc:513] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.346033 21390 catalog_manager.cc:5526] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
W20250112 02:57:24.348148 21378 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 36495fb158a440309eddedd3910a7ad3 consensus state has no leader
I20250112 02:57:24.350268 21385 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 170) took 2064 ms. Trace:
I20250112 02:57:24.350586 21385 rpcz_store.cc:276] 0112 02:57:22.285709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.285849 (+ 140us) service_pool.cc:225] Handling call
0112 02:57:22.707287 (+421438us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d8986 00007f7eed145644 00007f7eed1455d2 00007f7eed145501 00007f7eed14547c 00007f7eed144732 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:22.708168 (+ 881us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:22.708921 (+ 753us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:22.709638 (+ 717us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:22.709860 (+ 222us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:23.155138 (+445278us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:24.350210 (+1195072us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:23.145488 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:23.145835 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:23.145857 (+ 22us) write_op.cc:435] Acquired schema lock
0112 02:57:23.145870 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:23.146172 (+ 302us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:23.146208 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:23.146228 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:23.146359 (+ 131us) tablet.cc:698] Row locks acquired
0112 02:57:23.146383 (+ 24us) write_op.cc:260] PREPARE: finished
0112 02:57:23.146479 (+ 96us) write_op.cc:270] Start()
0112 02:57:23.146572 (+ 93us) write_op.cc:276] Timestamp: P: 1736650643146544 usec, L: 0
0112 02:57:23.146586 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:23.147004 (+ 418us) log.cc:844] Serialized 405 byte log entry
0112 02:57:23.152523 (+ 5519us) op_driver.cc:464] REPLICATION: finished
0112 02:57:23.153456 (+ 933us) write_op.cc:301] APPLY: starting
0112 02:57:23.153508 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:23.153597 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:23.153705 (+ 108us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:23.153874 (+ 169us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:23.153987 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:23.154024 (+ 37us) write_op.cc:312] APPLY: finished
0112 02:57:23.154169 (+ 145us) log.cc:844] Serialized 23 byte log entry
0112 02:57:23.154826 (+ 657us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.154897 (+ 71us) write_op.cc:454] Released schema lock
0112 02:57:23.154934 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:23.179414 (+ 24480us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.179437 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":967663488,"child_traces":[["op",{"apply.queue_time_us":786,"apply.run_cpu_time_us":1204,"apply.run_wall_time_us":1968,"num_ops":1,"prepare.queue_time_us":418269,"prepare.run_cpu_time_us":1792,"prepare.run_wall_time_us":33915,"raft.queue_time_us":163,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":573,"replication_time_us":5837}]]}
I20250112 02:57:24.357097 22145 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:24.357518 22145 ts_tablet_manager.cc:1397] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.276s user 0.007s sys 0.011s
I20250112 02:57:24.359616 22145 raft_consensus.cc:357] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.361992 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_842b05cd95284f1194954ffccb2e7bd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:24.363356 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_842b05cd95284f1194954ffccb2e7bd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:24.364207 22528 raft_consensus.cc:491] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.364637 22528 raft_consensus.cc:513] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.384136 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_531aa2f73c974001b166cf2852e82831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:24.368383 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 97) took 3337 ms. Trace:
I20250112 02:57:24.387948 22379 rpcz_store.cc:276] 0112 02:57:21.031113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.034635 (+ 3522us) service_pool.cc:225] Handling call
0112 02:57:24.010046 (+2975411us) ts_tablet_manager.cc:1145] Deleting tablet f9bb62ab3f274c29b2c5f9528890b816
0112 02:57:24.010082 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.119501 (+109419us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.215831 (+ 96330us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.368332 (+152501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2974889}
I20250112 02:57:24.388885 22379 tablet_replica.cc:331] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:24.389303 21387 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669] alter complete (version 1)
I20250112 02:57:24.389571 22379 raft_consensus.cc:2238] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:24.389938 22379 raft_consensus.cc:2267] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.391463 22379 ts_tablet_manager.cc:1905] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.382356 22145 raft_consensus.cc:383] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.394814 22145 raft_consensus.cc:738] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.373600 22143 tablet_bootstrap.cc:654] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.370381 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:24.366464 22528 leader_election.cc:290] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:24.403805 22145 consensus_queue.cc:260] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.377456 22043 raft_consensus.cc:2463] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:24.418325 21392 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab] alter complete (version 1)
I20250112 02:57:24.419471 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet f9bb62ab3f274c29b2c5f9528890b816 (table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]) successfully deleted
I20250112 02:57:24.423785 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:24.424865 22145 ts_tablet_manager.cc:1428] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.067s user 0.005s sys 0.000s
W20250112 02:57:24.425837 21978 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:24.426034 22145 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:24.428922 21978 consensus_peers.cc:487] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:24.436257 22041 raft_consensus.cc:2388] T 7af7e02e5fc248839b434f42f95c6509 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:24.436093 21690 leader_election.cc:304] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:24.436868 22528 raft_consensus.cc:2798] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:24.437283 22528 raft_consensus.cc:695] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:24.440392 22528 consensus_queue.cc:237] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.441902 22145 tablet_bootstrap.cc:654] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.434278 22645 consensus_queue.cc:1035] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250112 02:57:24.447523 22011 tablet_service.cc:1467] Processing CreateTablet for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.449915 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.454881 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 99e81199f79d46d6922eb1c6805d25da
I20250112 02:57:24.456292 21383 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 176) took 1254 ms. Trace:
I20250112 02:57:24.456841 22376 ts_tablet_manager.cc:1918] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:24.457162 22376 log.cc:1198] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:24.457868 22376 ts_tablet_manager.cc:1939] T 12f556cd7d6f4e13a525d3b8d4207d94 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:24.454823 21391 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143] alter complete (version 1)
I20250112 02:57:24.464412 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 (DEFAULT_TABLE table=default.Table_102d4e698b854093aa5caa6c1e25ad50 [id=49e7e77e80814c8f9128ea9878a6eb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.465535 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:24.465927 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 26) took 3033 ms (client timeout 1000 ms). Trace:
I20250112 02:57:24.456617 21383 rpcz_store.cc:276] 0112 02:57:23.202009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.303350 (+101341us) service_pool.cc:225] Handling call
0112 02:57:23.311195 (+ 7845us) spinlock_profiling.cc:243] Waited 7.13 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d8616 00007f7eed140b44 00007f7eed140ad2 00007f7eed140a01 00007f7eed14097c 00007f7eed13fc32 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:23.312236 (+ 1041us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:23.312739 (+ 503us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:23.312840 (+ 101us) catalog_manager.cc:2762] Locking tablets
0112 02:57:23.588611 (+275771us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:23.842450 (+253839us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:23.842675 (+ 225us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:24.454777 (+612102us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:24.456234 (+ 1457us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:23.694254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:23.694636 (+ 382us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:23.694659 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:23.694677 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:23.695066 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:23.695102 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:23.695128 (+ 26us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:23.695411 (+ 283us) tablet.cc:698] Row locks acquired
0112 02:57:23.695425 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:23.695520 (+ 95us) write_op.cc:270] Start()
0112 02:57:23.695584 (+ 64us) write_op.cc:276] Timestamp: P: 1736650643695605 usec, L: 0
0112 02:57:23.695597 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:23.797444 (+101847us) log.cc:844] Serialized 734 byte log entry
0112 02:57:23.841069 (+ 43625us) op_driver.cc:464] REPLICATION: finished
0112 02:57:23.841274 (+ 205us) write_op.cc:301] APPLY: starting
0112 02:57:23.841330 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:23.841425 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:23.841517 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:23.841746 (+ 229us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:23.841874 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:23.841928 (+ 54us) write_op.cc:312] APPLY: finished
0112 02:57:23.842073 (+ 145us) log.cc:844] Serialized 29 byte log entry
0112 02:57:23.842188 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.842251 (+ 63us) write_op.cc:454] Released schema lock
0112 02:57:23.842276 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:23.866102 (+ 23826us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.866127 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":16395648,"child_traces":[["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":1183,"apply.run_wall_time_us":5338,"mutex_wait_us":79877,"num_ops":2,"prepare.queue_time_us":103832,"prepare.run_cpu_time_us":2016,"prepare.run_wall_time_us":171811,"raft.queue_time_us":537,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":211,"replication_time_us":145385}]]}
I20250112 02:57:24.468842 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_732417b2b7f1413782c92f4849089b46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:24.471647 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_732417b2b7f1413782c92f4849089b46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:24.466220 21391 rpcz_store.cc:269] 0112 02:57:21.432604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.483271 (+ 50667us) service_pool.cc:225] Handling call
0112 02:57:24.465864 (+2982593us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:23.222686 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:23.222972 (+ 286us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:23.222990 (+ 18us) write_op.cc:435] Acquired schema lock
0112 02:57:23.223001 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:23.223196 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:23.223229 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:23.223247 (+ 18us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:23.223358 (+ 111us) tablet.cc:698] Row locks acquired
0112 02:57:23.223368 (+ 10us) write_op.cc:260] PREPARE: finished
0112 02:57:23.223456 (+ 88us) write_op.cc:270] Start()
0112 02:57:23.223528 (+ 72us) write_op.cc:276] Timestamp: P: 1736650643223553 usec, L: 0
0112 02:57:23.223541 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:23.223942 (+ 401us) log.cc:844] Serialized 356 byte log entry
0112 02:57:23.230156 (+ 6214us) op_driver.cc:464] REPLICATION: finished
0112 02:57:23.230457 (+ 301us) write_op.cc:301] APPLY: starting
0112 02:57:23.230523 (+ 66us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:23.230630 (+ 107us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:23.230734 (+ 104us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:23.230879 (+ 145us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:23.231026 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:23.231068 (+ 42us) write_op.cc:312] APPLY: finished
0112 02:57:23.231227 (+ 159us) log.cc:844] Serialized 23 byte log entry
0112 02:57:23.231356 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.231415 (+ 59us) write_op.cc:454] Released schema lock
0112 02:57:23.231445 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:23.245851 (+ 14406us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:23.245874 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9031,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":186,"apply.run_cpu_time_us":1147,"apply.run_wall_time_us":1147,"num_ops":1,"prepare.queue_time_us":475703,"prepare.run_cpu_time_us":1614,"prepare.run_wall_time_us":23149,"raft.queue_time_us":4581,"raft.run_cpu_time_us":594,"raft.run_wall_time_us":1207,"replication_time_us":6542}]]}
I20250112 02:57:24.484846 21391 master_service.cc:946] ReplaceTablet: received request to replace tablet 96f98f6840e24f3c9d03937c1bbc00e7 from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:24.504550 21387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 28) took 796 ms (client timeout 1000 ms). Trace:
W20250112 02:57:24.505084 21387 rpcz_store.cc:269] 0112 02:57:23.708276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.810451 (+102175us) service_pool.cc:225] Handling call
0112 02:57:24.504500 (+694049us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:24.281077 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:24.281459 (+ 382us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:24.281486 (+ 27us) write_op.cc:435] Acquired schema lock
0112 02:57:24.281500 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:24.281773 (+ 273us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:24.281812 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:24.281836 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:24.281976 (+ 140us) tablet.cc:698] Row locks acquired
0112 02:57:24.281990 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:24.282093 (+ 103us) write_op.cc:270] Start()
0112 02:57:24.282181 (+ 88us) write_op.cc:276] Timestamp: P: 1736650644282166 usec, L: 0
0112 02:57:24.282196 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:24.282696 (+ 500us) log.cc:844] Serialized 356 byte log entry
0112 02:57:24.290224 (+ 7528us) op_driver.cc:464] REPLICATION: finished
0112 02:57:24.290563 (+ 339us) write_op.cc:301] APPLY: starting
0112 02:57:24.290614 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:24.290714 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:24.290799 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:24.291501 (+ 702us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:24.291620 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:24.291663 (+ 43us) write_op.cc:312] APPLY: finished
0112 02:57:24.291832 (+ 169us) log.cc:844] Serialized 23 byte log entry
0112 02:57:24.291954 (+ 122us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.292008 (+ 54us) write_op.cc:454] Released schema lock
0112 02:57:24.292035 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:24.292243 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.292265 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25881,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":1684,"apply.run_wall_time_us":1689,"num_ops":1,"prepare.queue_time_us":10295,"prepare.run_cpu_time_us":1878,"prepare.run_wall_time_us":1884,"raft.queue_time_us":1063,"raft.run_cpu_time_us":611,"raft.run_wall_time_us":611,"replication_time_us":7945}]]}
W20250112 02:57:24.489919 21384 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 35) took 1605 ms (client timeout 1000 ms). Trace:
W20250112 02:57:24.495867 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 31) took 3448 ms (client timeout 1000 ms). Trace:
I20250112 02:57:24.500705 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_518c7dc00f1e44d0bb817840793eda43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:24.510413 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_518c7dc00f1e44d0bb817840793eda43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:24.511152 21384 rpcz_store.cc:269] 0112 02:57:22.884637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.992680 (+108043us) service_pool.cc:225] Handling call
0112 02:57:23.065631 (+ 72951us) spinlock_profiling.cc:243] Waited 60 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:24.489843 (+1424212us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":137942272}
I20250112 02:57:24.513494 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e3890251e644c1eb596de043b63491e from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:24.450397 22379 ts_tablet_manager.cc:1918] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:24.513897 22379 log.cc:1198] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:24.514622 22379 ts_tablet_manager.cc:1939] T 4166dcd1f9d74255b21810dd42b7358d P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
W20250112 02:57:24.514833 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_531aa2f73c974001b166cf2852e82831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:24.516196 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 32) took 1909 ms (client timeout 1000 ms). Trace:
I20250112 02:57:24.516391 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 106) took 3334 ms. Trace:
I20250112 02:57:24.517001 22143 tablet_bootstrap.cc:492] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:24.517429 22143 ts_tablet_manager.cc:1397] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.280s user 0.012s sys 0.006s
W20250112 02:57:24.516620 21386 rpcz_store.cc:269] 0112 02:57:22.606716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.742150 (+135434us) service_pool.cc:225] Handling call
0112 02:57:22.911733 (+169583us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:24.516123 (+1604390us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18513,"reported_tablets":18,"spinlock_wait_cycles":372544256}
I20250112 02:57:24.520526 22143 raft_consensus.cc:357] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.521144 22143 raft_consensus.cc:383] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.521373 22143 raft_consensus.cc:738] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:24.524451 21388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 27) took 1943 ms (client timeout 1000 ms). Trace:
I20250112 02:57:24.525235 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e81199f79d46d6922eb1c6805d25da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.525523 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f556cd7d6f4e13a525d3b8d4207d94 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:24.525115 21388 rpcz_store.cc:269] 0112 02:57:22.581256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.742070 (+160814us) service_pool.cc:225] Handling call
0112 02:57:22.905849 (+163779us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:24.524391 (+1618542us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":18,"spinlock_wait_cycles":365839104}
I20250112 02:57:24.526691 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.527879 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet bb324cd3120e45449e0500c99338135d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:15 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:24.508786 21389 rpcz_store.cc:269] 0112 02:57:21.047818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.175524 (+127706us) service_pool.cc:225] Handling call
0112 02:57:21.237172 (+ 61648us) spinlock_profiling.cc:243] Waited 53.1 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:24.495803 (+3258631us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:22.003998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:22.004316 (+ 318us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:22.004335 (+ 19us) write_op.cc:435] Acquired schema lock
0112 02:57:22.004346 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:22.004580 (+ 234us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:22.004607 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:22.004626 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:22.004863 (+ 237us) tablet.cc:698] Row locks acquired
0112 02:57:22.004875 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:22.004963 (+ 88us) write_op.cc:270] Start()
0112 02:57:22.005032 (+ 69us) write_op.cc:276] Timestamp: P: 1736650642005000 usec, L: 0
0112 02:57:22.005051 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:22.005468 (+ 417us) log.cc:844] Serialized 919 byte log entry
0112 02:57:22.056504 (+ 51036us) op_driver.cc:464] REPLICATION: finished
0112 02:57:22.060378 (+ 3874us) write_op.cc:301] APPLY: starting
0112 02:57:22.060439 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:22.060559 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:22.060668 (+ 109us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:22.060949 (+ 281us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:22.061108 (+ 159us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:22.061165 (+ 57us) write_op.cc:312] APPLY: finished
0112 02:57:22.061367 (+ 202us) log.cc:844] Serialized 29 byte log entry
0112 02:57:22.061520 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:22.061604 (+ 84us) write_op.cc:454] Released schema lock
0112 02:57:22.061639 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:22.070688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:22.071038 (+ 350us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:22.071058 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:22.071069 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:22.071287 (+ 218us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:22.071323 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:22.071341 (+ 18us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:22.071486 (+ 145us) tablet.cc:698] Row locks acquired
0112 02:57:22.071499 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:22.071596 (+ 97us) write_op.cc:270] Start()
0112 02:57:22.071668 (+ 72us) write_op.cc:276] Timestamp: P: 1736650642071659 usec, L: 0
0112 02:57:22.071684 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:22.149488 (+ 77804us) log.cc:844] Serialized 356 byte log entry
0112 02:57:22.150067 (+ 579us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b66d1 00007f7eed62d198 00007f7eeda993b1 00007f7eeda98b88 00007f7eeda9d821 00007f7eeda9d7d9 00007f7eeda9d769 00007f7eeda9d731 00007f7eeda9c95d 000000000059ea04 000000000059e839
0112 02:57:22.714894 (+564827us) op_driver.cc:464] REPLICATION: finished
0112 02:57:22.732830 (+ 17936us) write_op.cc:301] APPLY: starting
0112 02:57:22.732896 (+ 66us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:22.732979 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:22.732998 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:22.733155 (+ 157us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:22.733275 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:22.733315 (+ 40us) write_op.cc:312] APPLY: finished
0112 02:57:22.733477 (+ 162us) log.cc:844] Serialized 23 byte log entry
0112 02:57:22.733614 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:22.733672 (+ 58us) write_op.cc:454] Released schema lock
0112 02:57:22.733701 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":7808,"reported_tablets":17,"spinlock_wait_cycles":122020096,"child_traces":[["op",{"apply.queue_time_us":3750,"apply.run_cpu_time_us":1475,"apply.run_wall_time_us":52703,"mutex_wait_us":4017,"num_ops":2,"prepare.queue_time_us":4123,"prepare.run_cpu_time_us":1678,"prepare.run_wall_time_us":2525,"raft.queue_time_us":148,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":51383,"thread_start_us":75059,"threads_started":1}],["op",{"apply.queue_time_us":17769,"apply.run_cpu_time_us":1069,"apply.run_wall_time_us":1070,"num_ops":1,"prepare.queue_time_us":2143,"prepare.run_cpu_time_us":3292,"prepare.run_wall_time_us":79550,"raft.queue_time_us":2083,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":643096,"spinlock_wait_cycles":177573632,"thread_start_us":18922,"threads_started":1}]]}
I20250112 02:57:24.516692 22379 rpcz_store.cc:276] 0112 02:57:21.181527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.181658 (+ 131us) service_pool.cc:225] Handling call
0112 02:57:24.388757 (+3207099us) ts_tablet_manager.cc:1145] Deleting tablet 4166dcd1f9d74255b21810dd42b7358d
0112 02:57:24.388793 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.397883 (+ 9090us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.450212 (+ 52329us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.516322 (+ 66110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3206501}
I20250112 02:57:24.535274 22379 tablet_replica.cc:331] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:24.535995 22379 raft_consensus.cc:2238] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:24.536383 22379 raft_consensus.cc:2267] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.560184 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_582c4ae70bd94161908109db890c099e" } new_table_name: "default.Table_ae5cff4a4f3e4918b0f33825d66c8092" modify_external_catalogs: true
I20250112 02:57:24.570756 21389 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e2cfbf3f568b436f8d819d76e84c68e8" } new_table_name: "default.Table_3c00d9bac968444ea42965a2f7222461" modify_external_catalogs: true
W20250112 02:57:24.575409 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250112 02:57:24.575510 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_84ec605ca7bd4c48b8fc9e590652654c" } modify_external_catalogs: true
I20250112 02:57:24.576297 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_84ec605ca7bd4c48b8fc9e590652654c" } modify_external_catalogs: true
I20250112 02:57:24.586349 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_4ebef261df1e4307b326cfa321547410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:24.588181 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ebef261df1e4307b326cfa321547410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:24.618178 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]) successfully deleted
W20250112 02:57:24.618924 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f7700b8cce4261a32f2009d1481b99 already in progress: creating tablet
I20250112 02:57:24.555040 22143 consensus_queue.cc:260] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.592746 22379 ts_tablet_manager.cc:1905] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.628712 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 5890ebb3c55943d5a334d6c93fbdaf0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:16 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.635368 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 5e3890251e644c1eb596de043b63491e
I20250112 02:57:24.636399 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 402c24e54ab040e29949b1e4efbe5ff3 with delete_type TABLET_DATA_DELETED (replaced by tablet c48d1bc147ba48ebbebfd707dc44fe02) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.636859 21387 catalog_manager.cc:6596] ReplaceTablet: tablet 5e3890251e644c1eb596de043b63491e deleted and replaced by tablet 78f03a1dfb604b33a0528c28ade287f2
I20250112 02:57:24.639216 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 4166dcd1f9d74255b21810dd42b7358d (table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]) successfully deleted
I20250112 02:57:24.529837 22568 leader_election.cc:290] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.634236 22020 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e81199f79d46d6922eb1c6805d25da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:24.657133 22143 ts_tablet_manager.cc:1428] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.139s user 0.006s sys 0.007s
I20250112 02:57:24.657518 21882 tablet_service.cc:1467] Processing CreateTablet for tablet a69123c4ab384e0380c62e52b9f3b42d (DEFAULT_TABLE table=default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0 [id=8dbf245ab5354db7a93b54749e978c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.657955 22143 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:24.658428 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a69123c4ab384e0380c62e52b9f3b42d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.652720 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:24.662503 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.663491 22143 tablet_bootstrap.cc:654] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.655684 22568 raft_consensus.cc:491] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.663314 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:24.669663 21384 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:24.669056 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 (DEFAULT_TABLE table=default.Table_102d4e698b854093aa5caa6c1e25ad50 [id=49e7e77e80814c8f9128ea9878a6eb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.670698 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.675136 22145 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:24.675653 22145 ts_tablet_manager.cc:1397] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.250s user 0.010s sys 0.000s
I20250112 02:57:24.681699 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 (DEFAULT_TABLE table=default.Table_102d4e698b854093aa5caa6c1e25ad50 [id=49e7e77e80814c8f9128ea9878a6eb21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.682767 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.685046 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e81199f79d46d6922eb1c6805d25da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.685735 22375 tablet_replica.cc:331] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:24.686414 22375 raft_consensus.cc:2238] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:24.687288 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:24.688654 22143 tablet_bootstrap.cc:492] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:24.689014 22143 ts_tablet_manager.cc:1397] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20250112 02:57:24.619763 22585 raft_consensus.cc:695] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:24.691759 22143 raft_consensus.cc:357] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.692359 22143 raft_consensus.cc:383] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.692570 22143 raft_consensus.cc:738] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.693106 22143 consensus_queue.cc:260] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.695542 22143 ts_tablet_manager.cc:1428] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 02:57:24.696281 22143 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:24.700647 21387 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_77fe18e65c724b129a710891e2e7e123" } modify_external_catalogs: true
I20250112 02:57:24.701371 22143 tablet_bootstrap.cc:654] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.701289 21387 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_77fe18e65c724b129a710891e2e7e123" } modify_external_catalogs: true
I20250112 02:57:24.704946 22144 tablet_bootstrap.cc:492] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:24.705350 22144 ts_tablet_manager.cc:1397] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.569s user 0.010s sys 0.029s
W20250112 02:57:24.715612 21389 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2ce633c5b3b4cbea17c171564fa72d6 consensus state has no leader
I20250112 02:57:24.717263 22642 raft_consensus.cc:491] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.717595 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:24.724866 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2601daad8ba742ec88ccfe603440e8e7 consensus state has no leader
I20250112 02:57:24.727114 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:24.729672 21391 catalog_manager.cc:6596] ReplaceTablet: tablet 96f98f6840e24f3c9d03937c1bbc00e7 deleted and replaced by tablet 2d793344526a404e8a737e7fb628e521
I20250112 02:57:24.730123 21391 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50642 (request call id 184) took 1068 ms. Trace:
I20250112 02:57:24.730556 21391 rpcz_store.cc:276] 0112 02:57:23.661630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.484648 (+823018us) service_pool.cc:225] Handling call
0112 02:57:24.730061 (+245413us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:24.667149 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:24.667434 (+ 285us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:24.667453 (+ 19us) write_op.cc:435] Acquired schema lock
0112 02:57:24.667464 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:24.667717 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:24.667744 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:24.667762 (+ 18us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:24.667903 (+ 141us) tablet.cc:698] Row locks acquired
0112 02:57:24.667913 (+ 10us) write_op.cc:260] PREPARE: finished
0112 02:57:24.667992 (+ 79us) write_op.cc:270] Start()
0112 02:57:24.668053 (+ 61us) write_op.cc:276] Timestamp: P: 1736650644668054 usec, L: 0
0112 02:57:24.668065 (+ 12us) op_driver.cc:348] REPLICATION: starting
0112 02:57:24.699385 (+ 31320us) log.cc:844] Serialized 631 byte log entry
0112 02:57:24.718522 (+ 19137us) op_driver.cc:464] REPLICATION: finished
0112 02:57:24.720689 (+ 2167us) write_op.cc:301] APPLY: starting
0112 02:57:24.720754 (+ 65us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:24.720881 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:24.721011 (+ 130us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:24.721406 (+ 395us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:24.721652 (+ 246us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:24.721710 (+ 58us) write_op.cc:312] APPLY: finished
0112 02:57:24.721902 (+ 192us) log.cc:844] Serialized 29 byte log entry
0112 02:57:24.722036 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.722106 (+ 70us) write_op.cc:454] Released schema lock
0112 02:57:24.722140 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:24.722370 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.722402 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2030,"apply.run_cpu_time_us":1642,"apply.run_wall_time_us":1641,"num_ops":2,"prepare.queue_time_us":80594,"prepare.run_cpu_time_us":1844,"prepare.run_wall_time_us":44515,"raft.queue_time_us":126,"raft.run_cpu_time_us":13,"raft.run_wall_time_us":13,"replication_time_us":50361}]]}
I20250112 02:57:24.721235 22642 raft_consensus.cc:513] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.738054 22642 leader_election.cc:290] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.739293 21902 raft_consensus.cc:3054] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.744055 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.719642 21875 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:24.749089 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 109277us
I20250112 02:57:24.749172 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.757920 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.666286 22524 consensus_queue.cc:1035] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.072s
I20250112 02:57:24.748915 21902 raft_consensus.cc:2463] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:24.764873 22640 consensus_queue.cc:1159] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20250112 02:57:24.762895 21874 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:24.771852 21980 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:24.772485 21877 consensus_meta.cc:287] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent flushing consensus metadata: real 0.513s user 0.005s sys 0.000s
I20250112 02:57:24.772333 21980 leader_election.cc:304] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:24.773442 22375 raft_consensus.cc:2267] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.773694 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b6864a288e5f4be18484745f91242d64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:24.774941 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7af7e02e5fc248839b434f42f95c6509" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:24.775678 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6864a288e5f4be18484745f91242d64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:24.778856 21691 leader_election.cc:304] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:24.779529 22528 raft_consensus.cc:3054] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:24.720659 21876 tablet_service.cc:1467] Processing CreateTablet for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (DEFAULT_TABLE table=default.Table_50bcfd9aa201416e976e67c56aa7f3f6 [id=8a4d6d88d54a46c4964157ad79415e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:24.780570 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:24.784264 22144 raft_consensus.cc:357] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.785948 22528 raft_consensus.cc:2743] T 7af7e02e5fc248839b434f42f95c6509 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:24.784899 22144 raft_consensus.cc:383] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.786592 22144 raft_consensus.cc:738] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.787091 22144 consensus_queue.cc:260] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.789755 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:24.791258 21385 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 182) took 1225 ms. Trace:
I20250112 02:57:24.791610 21385 rpcz_store.cc:276] 0112 02:57:23.566332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.383704 (+817372us) service_pool.cc:225] Handling call
0112 02:57:24.515408 (+131704us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:24.588446 (+ 73038us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:24.790674 (+202228us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:24.790977 (+ 303us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:24.791208 (+ 231us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:24.711820 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:24.712229 (+ 409us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:24.712259 (+ 30us) write_op.cc:435] Acquired schema lock
0112 02:57:24.712277 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:24.712603 (+ 326us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:24.712646 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:24.712684 (+ 38us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:24.718064 (+ 5380us) tablet.cc:698] Row locks acquired
0112 02:57:24.718098 (+ 34us) write_op.cc:260] PREPARE: finished
0112 02:57:24.718229 (+ 131us) write_op.cc:270] Start()
0112 02:57:24.718326 (+ 97us) write_op.cc:276] Timestamp: P: 1736650644718295 usec, L: 0
0112 02:57:24.718347 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 02:57:24.734486 (+ 16139us) log.cc:844] Serialized 433 byte log entry
0112 02:57:24.750821 (+ 16335us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b66d1 00007f7eed62d198 00007f7eeda993b1 00007f7eeda98b88 00007f7eeda9d821 00007f7eeda9d7d9 00007f7eeda9d769 00007f7eeda9d731 00007f7eeda9c95d 000000000059ea04 000000000059e839
0112 02:57:24.788225 (+ 37404us) op_driver.cc:464] REPLICATION: finished
0112 02:57:24.789396 (+ 1171us) write_op.cc:301] APPLY: starting
0112 02:57:24.789444 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:24.789532 (+ 88us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:24.789548 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:24.789883 (+ 335us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:24.790023 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:24.790061 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:24.790222 (+ 161us) log.cc:844] Serialized 29 byte log entry
0112 02:57:24.790379 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.790443 (+ 64us) write_op.cc:454] Released schema lock
0112 02:57:24.790475 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:24.790725 (+ 250us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.790748 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1008,"apply.run_cpu_time_us":1296,"apply.run_wall_time_us":1296,"num_ops":2,"prepare.queue_time_us":122393,"prepare.run_cpu_time_us":4086,"prepare.run_wall_time_us":74087,"raft.queue_time_us":62571,"raft.run_cpu_time_us":710,"raft.run_wall_time_us":889,"replication_time_us":69782,"spinlock_wait_cycles":35757824}]]}
I20250112 02:57:24.794909 22144 ts_tablet_manager.cc:1428] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.088s user 0.003s sys 0.003s
I20250112 02:57:24.795590 22144 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:24.798446 22645 consensus_queue.cc:1035] T aab2cf8bc28b47239eaa3dcaa4c90e9c P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:24.796999 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 94) took 4714 ms. Trace:
I20250112 02:57:24.801029 22376 rpcz_store.cc:276] 0112 02:57:20.082154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.083742 (+ 1588us) service_pool.cc:225] Handling call
0112 02:57:23.817952 (+3734210us) ts_tablet_manager.cc:1145] Deleting tablet 12f556cd7d6f4e13a525d3b8d4207d94
0112 02:57:23.818002 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.112788 (+294786us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.456647 (+343859us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.796931 (+340284us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":621,"tablet-delete.queue_time_us":3733614}
I20250112 02:57:24.802692 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 100) took 4583 ms. Trace:
I20250112 02:57:24.803028 22376 rpcz_store.cc:276] 0112 02:57:20.219048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.251093 (+ 32045us) service_pool.cc:225] Handling call
0112 02:57:24.802117 (+4551024us) ts_tablet_manager.cc:1145] Deleting tablet bb324cd3120e45449e0500c99338135d
0112 02:57:24.802177 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.802643 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550488}
I20250112 02:57:24.808284 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.808784 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e589965a6e364011b9d11bea87be94c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:24.809360 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.810287 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.811288 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:24.812508 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e589965a6e364011b9d11bea87be94c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:24.809684 22652 raft_consensus.cc:491] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.810094 22655 raft_consensus.cc:2743] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:24.815042 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 103) took 4559 ms. Trace:
I20250112 02:57:24.816349 22376 rpcz_store.cc:276] 0112 02:57:20.255370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.255455 (+ 85us) service_pool.cc:225] Handling call
0112 02:57:24.810472 (+4555017us) ts_tablet_manager.cc:1145] Deleting tablet 402c24e54ab040e29949b1e4efbe5ff3
0112 02:57:24.814824 (+ 4352us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.814998 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4554517}
I20250112 02:57:24.819381 22585 consensus_queue.cc:237] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.816108 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:24.745832 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:24.825630 22375 ts_tablet_manager.cc:1905] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.827425 22143 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:24.827800 22143 ts_tablet_manager.cc:1397] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.015s
I20250112 02:57:24.829811 22143 raft_consensus.cc:357] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.830408 22143 raft_consensus.cc:383] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.830673 22143 raft_consensus.cc:738] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.834213 22143 consensus_queue.cc:260] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.836098 22143 ts_tablet_manager.cc:1428] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250112 02:57:24.842227 22143 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:24.849164 22143 tablet_bootstrap.cc:654] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.854661 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166dcd1f9d74255b21810dd42b7358d with delete_type TABLET_DATA_DELETED (replaced by tablet e21a0e4b5ebc426383cf332f46492b80) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.681223 22568 raft_consensus.cc:513] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.863202 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 101) took 4610 ms. Trace:
I20250112 02:57:24.863466 22376 rpcz_store.cc:276] 0112 02:57:20.252435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:20.257588 (+ 5153us) service_pool.cc:225] Handling call
0112 02:57:24.824883 (+4567295us) ts_tablet_manager.cc:1145] Deleting tablet 35b39d0cf44f4e91908b8e568df0a16a
0112 02:57:24.862939 (+ 38056us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.863147 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4566776}
I20250112 02:57:24.864331 22376 tablet_replica.cc:331] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:24.874132 22145 raft_consensus.cc:357] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.874786 22145 raft_consensus.cc:383] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:24.875007 22145 raft_consensus.cc:738] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:24.875654 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_518c7dc00f1e44d0bb817840793eda43" } modify_external_catalogs: true
I20250112 02:57:24.875497 22145 consensus_queue.cc:260] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.876114 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_518c7dc00f1e44d0bb817840793eda43" } modify_external_catalogs: true
I20250112 02:57:24.871243 22652 raft_consensus.cc:513] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:24.879917 22652 leader_election.cc:290] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:24.880174 22524 consensus_queue.cc:1035] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:24.893275 22379 ts_tablet_manager.cc:1918] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:24.893730 22379 log.cc:1198] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:24.894433 22379 ts_tablet_manager.cc:1939] T c48d1bc147ba48ebbebfd707dc44fe02 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:24.895213 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 110) took 3050 ms. Trace:
I20250112 02:57:24.895534 22379 rpcz_store.cc:276] 0112 02:57:21.845130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.846514 (+ 1384us) service_pool.cc:225] Handling call
0112 02:57:24.535132 (+2688618us) ts_tablet_manager.cc:1145] Deleting tablet c48d1bc147ba48ebbebfd707dc44fe02
0112 02:57:24.535174 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.751249 (+216075us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.893058 (+141809us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:24.895169 (+ 2111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2687765}
I20250112 02:57:24.896970 22379 tablet_replica.cc:331] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:24.898095 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:24.898631 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet c48d1bc147ba48ebbebfd707dc44fe02 (table default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]) successfully deleted
I20250112 02:57:24.900681 22144 tablet_bootstrap.cc:654] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:24.901926 22644 raft_consensus.cc:491] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:24.902323 22644 raft_consensus.cc:513] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:24.904184 22644 leader_election.cc:290] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:24.907987 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.908677 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.909333 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:24.914264 21391 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f60bc97e19974e32ab058969ab9a6986
I20250112 02:57:24.917800 22376 raft_consensus.cc:2238] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:24.919121 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ae5cff4a4f3e4918b0f33825d66c8092" } new_table_name: "default.Table_ae7a8479142d4028817982eb1bdf10c2" modify_external_catalogs: true
I20250112 02:57:24.920188 22376 raft_consensus.cc:2267] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.927995 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3f830b1a9d5e4df0b421de4dc7d6207b" } new_table_name: "default.Table_8324e31f290e4ea2b09e7dc7c6e76f51" modify_external_catalogs: true
I20250112 02:57:24.934840 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:24.936012 21691 leader_election.cc:343] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:24.942255 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 170510us
W20250112 02:57:24.943890 21577 debug-util.cc:398] Leaking SignalData structure 0x7b080002bc80 after lost signal to thread 21509
W20250112 02:57:24.944718 21577 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3120 after lost signal to thread 21580
I20250112 02:57:24.941924 22568 leader_election.cc:290] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:24.947569 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:24.948156 22044 raft_consensus.cc:2463] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:24.955124 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2601daad8ba742ec88ccfe603440e8e7 consensus state has no leader
I20250112 02:57:24.959594 22145 ts_tablet_manager.cc:1428] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.284s user 0.006s sys 0.000s
I20250112 02:57:24.960582 22145 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:24.934046 22379 raft_consensus.cc:2238] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:24.961133 22379 raft_consensus.cc:2267] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:24.963091 22379 ts_tablet_manager.cc:1905] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.965811 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 12f556cd7d6f4e13a525d3b8d4207d94 (table default.Table_02f0da840c0646d59412dcb267a4d571 [id=ed996933e39d472992d6f351ae51fb39]) successfully deleted
W20250112 02:57:24.971825 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0977a10252840a381234f02bb7339b3 consensus state has no leader
W20250112 02:57:24.972797 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet bb324cd3120e45449e0500c99338135d because the tablet was not found. No further retry: Not found: Tablet not found: bb324cd3120e45449e0500c99338135d
I20250112 02:57:24.973158 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:24.979081 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 35b39d0cf44f4e91908b8e568df0a16a because the tablet was not found. No further retry: Not found: Tablet not found: 35b39d0cf44f4e91908b8e568df0a16a
I20250112 02:57:24.982549 22376 ts_tablet_manager.cc:1905] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:24.983019 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48d1bc147ba48ebbebfd707dc44fe02" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:24.983704 21898 raft_consensus.cc:2463] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
W20250112 02:57:24.986886 21838 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.006s sys 0.018s
W20250112 02:57:24.987143 21838 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.551s user 0.006s sys 0.018s
I20250112 02:57:24.998006 22375 ts_tablet_manager.cc:1918] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:24.998245 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_2573b35f681a4ede8201db20c64d179e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:24.999912 22007 consensus_meta.cc:287] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent flushing consensus metadata: real 0.545s user 0.007s sys 0.000s
W20250112 02:57:25.000061 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2573b35f681a4ede8201db20c64d179e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.004518 21690 leader_election.cc:304] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.002660 22528 raft_consensus.cc:491] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.007244 22145 tablet_bootstrap.cc:654] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.006848 22528 raft_consensus.cc:513] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.008527 22528 leader_election.cc:290] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:25.006011 22644 raft_consensus.cc:2798] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:25.009160 22644 raft_consensus.cc:491] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:25.009474 22644 raft_consensus.cc:3054] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.011899 22663 raft_consensus.cc:491] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.012271 22663 raft_consensus.cc:513] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.014087 22663 leader_election.cc:290] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:25.014968 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ff93d70a357e442dafed35cd75ae6152"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:25.016328 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff93d70a357e442dafed35cd75ae6152 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.024394 21383 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_68a1e71fb38145f797bf5a1740443cb4" } modify_external_catalogs: true
I20250112 02:57:25.024986 21383 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_68a1e71fb38145f797bf5a1740443cb4" } modify_external_catalogs: true
I20250112 02:57:25.046540 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.048175 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_78a32d16e1b242c094697398cc472077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.048831 22145 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:25.049181 22145 ts_tablet_manager.cc:1397] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.000s
W20250112 02:57:25.049559 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78a32d16e1b242c094697398cc472077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.052332 22644 raft_consensus.cc:513] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.059890 22568 consensus_queue.cc:1035] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:25.061101 21753 raft_consensus.cc:3054] T f47e14720a704a19ac3aea8723e13c28 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.068064 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:25.071036 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_33d0ee945fe34221b3848101d41218cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.071691 22376 ts_tablet_manager.cc:1918] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:25.072026 22376 log.cc:1198] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/f9bb62ab3f274c29b2c5f9528890b816
W20250112 02:57:25.072531 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33d0ee945fe34221b3848101d41218cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.072664 22376 ts_tablet_manager.cc:1939] T f9bb62ab3f274c29b2c5f9528890b816 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:25.073239 22143 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:25.073890 22143 ts_tablet_manager.cc:1397] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.232s user 0.022s sys 0.004s
I20250112 02:57:25.078377 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.084482 22145 raft_consensus.cc:357] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.085198 22145 raft_consensus.cc:383] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.085465 22145 raft_consensus.cc:738] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.085783 22644 leader_election.cc:290] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:25.099727 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eef30508946e4376961b6cd463172c1f" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:25.100428 22044 raft_consensus.cc:2388] T eef30508946e4376961b6cd463172c1f P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:25.101787 21980 leader_election.cc:304] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:25.091588 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 87666.8us
I20250112 02:57:25.102983 22143 raft_consensus.cc:357] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.104612 22143 raft_consensus.cc:383] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.104852 22143 raft_consensus.cc:738] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.104269 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:25.105858 22044 raft_consensus.cc:2388] T 557de3de9a4d4d0bac1dc16f66ab6b01 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:25.105443 22143 consensus_queue.cc:260] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.105842 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:25.107182 22041 raft_consensus.cc:3054] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.009385 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:25.046422 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:25.068574 22144 tablet_bootstrap.cc:492] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:25.087055 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22aac4332d0e476d9758004972ef8ab2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:25.111665 22144 ts_tablet_manager.cc:1397] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.316s user 0.007s sys 0.002s
I20250112 02:57:25.114400 22669 raft_consensus.cc:2798] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:57:25.114995 21691 leader_election.cc:343] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.115470 22375 log.cc:1198] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/99e81199f79d46d6922eb1c6805d25da
I20250112 02:57:25.116683 22375 ts_tablet_manager.cc:1939] T 99e81199f79d46d6922eb1c6805d25da P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:25.118479 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7b171c9ac8164714bfeeb9226098e7ff" } modify_external_catalogs: true
I20250112 02:57:25.118994 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7b171c9ac8164714bfeeb9226098e7ff" } modify_external_catalogs: true
I20250112 02:57:25.119313 22669 raft_consensus.cc:491] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:25.119621 22669 raft_consensus.cc:3054] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.126628 22143 ts_tablet_manager.cc:1428] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20250112 02:57:25.127377 22143 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:25.127720 22379 ts_tablet_manager.cc:1918] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:25.130158 21836 leader_election.cc:304] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:25.131000 22379 log.cc:1198] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:25.133328 22653 raft_consensus.cc:2743] T eef30508946e4376961b6cd463172c1f P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:25.136864 22379 ts_tablet_manager.cc:1939] T 33a5c293bbd643aab9409c6e9abe4f0a P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:25.143086 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 112) took 3219 ms. Trace:
I20250112 02:57:25.148360 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 0bca1c68aae24a1f9974a0bd2244fbf8
I20250112 02:57:25.145457 22143 tablet_bootstrap.cc:654] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:25.150985 21980 leader_election.cc:343] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Not found: Tablet not found: 4166dcd1f9d74255b21810dd42b7358d
W20250112 02:57:25.152923 21978 leader_election.cc:343] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:25.086077 22145 consensus_queue.cc:260] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:25.151639 21980 consensus_peers.cc:487] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:25.151063 22379 rpcz_store.cc:276] 0112 02:57:21.924092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.924282 (+ 190us) service_pool.cc:225] Handling call
0112 02:57:24.896850 (+2972568us) ts_tablet_manager.cc:1145] Deleting tablet 33a5c293bbd643aab9409c6e9abe4f0a
0112 02:57:24.896883 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.075843 (+178960us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.102414 (+ 26571us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.143038 (+ 40624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2970220}
I20250112 02:57:25.157925 22145 ts_tablet_manager.cc:1428] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.109s user 0.004s sys 0.002s
I20250112 02:57:25.159365 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_309cdb1f2ef042d0a115d9709cf410ae" } new_table_name: "default.Table_3b67ac5b84644c469dcf959098497e51" modify_external_catalogs: true
I20250112 02:57:25.159937 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 105) took 3716 ms. Trace:
I20250112 02:57:25.160231 22376 rpcz_store.cc:276] 0112 02:57:21.443845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.444889 (+ 1044us) service_pool.cc:225] Handling call
0112 02:57:24.864202 (+3419313us) ts_tablet_manager.cc:1145] Deleting tablet f9bb62ab3f274c29b2c5f9528890b816
0112 02:57:24.864241 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:24.992254 (+128013us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.071489 (+ 79235us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.159890 (+ 88401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98915,"tablet-delete.queue_time_us":3269526}
I20250112 02:57:25.161250 22376 tablet_replica.cc:331] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:25.163314 22568 consensus_queue.cc:1035] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:25.166231 22379 tablet_replica.cc:331] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:25.166726 22653 consensus_queue.cc:1035] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:25.167549 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:25.166117 22524 consensus_queue.cc:1035] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250112 02:57:25.174326 22672 raft_consensus.cc:491] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.174921 22672 raft_consensus.cc:513] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.176611 22379 raft_consensus.cc:2238] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:25.176851 22672 leader_election.cc:290] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:25.177116 22379 raft_consensus.cc:2267] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.177770 22568 consensus_queue.cc:1035] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:25.178606 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_842b05cd95284f1194954ffccb2e7bd3" } new_table_name: "default.Table_f4f19feee3254ca69d7279dcaa65cb82" modify_external_catalogs: true
W20250112 02:57:25.179244 21980 leader_election.cc:343] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.183756 22143 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:25.184100 22143 ts_tablet_manager.cc:1397] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.011s
I20250112 02:57:25.186281 22143 raft_consensus.cc:357] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.186890 22143 raft_consensus.cc:383] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:25.186884 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 36) took 1286 ms (client timeout 1000 ms). Trace:
I20250112 02:57:25.186968 22145 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:25.187122 22143 raft_consensus.cc:738] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.187629 22143 consensus_queue.cc:260] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:25.187172 21390 rpcz_store.cc:269] 0112 02:57:23.900472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.253901 (+353429us) service_pool.cc:225] Handling call
0112 02:57:25.186821 (+932920us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:24.473931 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:24.474385 (+ 454us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:24.474417 (+ 32us) write_op.cc:435] Acquired schema lock
0112 02:57:24.474436 (+ 19us) tablet.cc:659] Decoding operations
0112 02:57:24.474781 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:24.474828 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:24.474847 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:24.474974 (+ 127us) tablet.cc:698] Row locks acquired
0112 02:57:24.474984 (+ 10us) write_op.cc:260] PREPARE: finished
0112 02:57:24.475078 (+ 94us) write_op.cc:270] Start()
0112 02:57:24.475167 (+ 89us) write_op.cc:276] Timestamp: P: 1736650644475058 usec, L: 0
0112 02:57:24.475187 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 02:57:24.490550 (+ 15363us) log.cc:844] Serialized 536 byte log entry
0112 02:57:24.499358 (+ 8808us) op_driver.cc:464] REPLICATION: finished
0112 02:57:24.500508 (+ 1150us) write_op.cc:301] APPLY: starting
0112 02:57:24.500575 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:24.500700 (+ 125us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:24.500822 (+ 122us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:24.501095 (+ 273us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:24.501266 (+ 171us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:24.501328 (+ 62us) write_op.cc:312] APPLY: finished
0112 02:57:24.501545 (+ 217us) log.cc:844] Serialized 23 byte log entry
0112 02:57:24.501723 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.501801 (+ 78us) write_op.cc:454] Released schema lock
0112 02:57:24.501840 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:24.502114 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:24.502149 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":946,"apply.run_cpu_time_us":1548,"apply.run_wall_time_us":1547,"num_ops":1,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":2261,"prepare.run_wall_time_us":19095,"raft.queue_time_us":102,"raft.run_cpu_time_us":657,"raft.run_wall_time_us":657,"replication_time_us":24055}]]}
I20250112 02:57:25.189550 22143 ts_tablet_manager.cc:1428] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250112 02:57:25.190297 22143 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:25.191797 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]) successfully deleted
I20250112 02:57:25.192067 22379 ts_tablet_manager.cc:1905] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.193766 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9543a0a0359e4b72bd30f1cb6290f767" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:25.197664 22143 tablet_bootstrap.cc:654] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.198781 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet f9bb62ab3f274c29b2c5f9528890b816 (table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]) successfully deleted
I20250112 02:57:25.161912 22376 raft_consensus.cc:2238] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:25.206620 22654 raft_consensus.cc:491] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:25.208074 21978 leader_election.cc:343] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.208454 21978 leader_election.cc:304] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:25.209434 21978 leader_election.cc:343] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:25.209770 21978 leader_election.cc:304] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.207052 22654 raft_consensus.cc:513] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.212388 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:25.212985 22654 leader_election.cc:290] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:25.213276 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 4166dcd1f9d74255b21810dd42b7358d with delete_type TABLET_DATA_DELETED (replaced by tablet e21a0e4b5ebc426383cf332f46492b80) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:25.211683 22457 raft_consensus.cc:2743] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:25.218365 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:25.219439 21691 leader_election.cc:343] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.228415 22041 raft_consensus.cc:2463] T 22aac4332d0e476d9758004972ef8ab2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:25.230131 22669 raft_consensus.cc:513] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.233278 22663 raft_consensus.cc:2701] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Received pre-election callback for term 2 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250112 02:57:25.234019 22663 consensus_queue.cc:1035] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:25.237424 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:25.242750 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:25.243072 22457 raft_consensus.cc:491] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.243431 22457 raft_consensus.cc:513] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.244957 22457 leader_election.cc:290] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:25.244943 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 138866us
I20250112 02:57:25.246589 22376 raft_consensus.cc:2267] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.248088 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.250248 22376 ts_tablet_manager.cc:1905] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.252687 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994e1d114e5a4de5aa1d7651907d718e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:25.253484 22143 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:25.253858 22143 ts_tablet_manager.cc:1397] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.064s user 0.010s sys 0.001s
I20250112 02:57:25.255892 22143 raft_consensus.cc:357] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.256502 22143 raft_consensus.cc:383] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.256726 22143 raft_consensus.cc:738] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.257239 22143 consensus_queue.cc:260] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.257840 21383 master_service.cc:946] ReplaceTablet: received request to replace tablet 443aa544f6df4d50a78c9effd306e216 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:25.261507 22143 ts_tablet_manager.cc:1428] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250112 02:57:25.262243 22143 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:25.269048 22143 tablet_bootstrap.cc:654] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.269766 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa732d27f144aaad088ecc072837e9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:25.272367 22689 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250112 02:57:25.279047 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e2b248224f9416faf4bd83a853ec918 consensus state has no leader
W20250112 02:57:25.287374 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4716677cfad74c0dafe18c55796aeae5 has no consensus state
I20250112 02:57:25.251276 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:25.291924 21980 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:25.292505 21980 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.254956 22684 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250112 02:57:25.284399 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 48313.3us
I20250112 02:57:25.114058 22144 raft_consensus.cc:357] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.297685 22144 raft_consensus.cc:383] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.298023 22144 raft_consensus.cc:738] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.298786 22144 consensus_queue.cc:260] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.282670 22669 leader_election.cc:290] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:25.312240 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9543a0a0359e4b72bd30f1cb6290f767" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:25.326243 22145 tablet_bootstrap.cc:654] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.328981 22379 ts_tablet_manager.cc:1918] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:25.329406 22379 log.cc:1198] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:25.329613 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48d1bc147ba48ebbebfd707dc44fe02" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:25.330233 21902 raft_consensus.cc:3054] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.330322 22379 ts_tablet_manager.cc:1939] T 5f787ffe65e34e8f9852d149fff9f947 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:25.332158 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 114) took 3345 ms. Trace:
I20250112 02:57:25.332450 22379 rpcz_store.cc:276] 0112 02:57:21.987077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.990617 (+ 3540us) service_pool.cc:225] Handling call
0112 02:57:25.166105 (+3175488us) ts_tablet_manager.cc:1145] Deleting tablet 5f787ffe65e34e8f9852d149fff9f947
0112 02:57:25.166140 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.239188 (+ 73048us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.328756 (+ 89568us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.332112 (+ 3356us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9540,"tablet-delete.queue_time_us":3174677}
I20250112 02:57:25.333539 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 117) took 3220 ms. Trace:
I20250112 02:57:25.333812 22379 rpcz_store.cc:276] 0112 02:57:22.113084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.113178 (+ 94us) service_pool.cc:225] Handling call
0112 02:57:25.333323 (+3220145us) ts_tablet_manager.cc:1145] Deleting tablet 11fbebff35c344bdb305ea1a6dbbefe4
0112 02:57:25.333359 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.333497 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219678}
I20250112 02:57:25.336174 21902 raft_consensus.cc:2463] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:25.340073 21980 leader_election.cc:304] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:25.341439 22581 raft_consensus.cc:2798] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:25.341897 22581 raft_consensus.cc:695] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:25.342664 22581 consensus_queue.cc:237] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:25.352053 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 77d80c6a60bc46449c7c0656128100c8 consensus state has no leader
I20250112 02:57:25.312886 22041 raft_consensus.cc:2463] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:25.353066 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c" } modify_external_catalogs: true
I20250112 02:57:25.353703 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c" } modify_external_catalogs: true
I20250112 02:57:25.360239 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_f12f747999f9485880eb1238ed2ecfd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.361265 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 118) took 3247 ms. Trace:
W20250112 02:57:25.361804 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f12f747999f9485880eb1238ed2ecfd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.361650 22379 rpcz_store.cc:276] 0112 02:57:22.114055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.114148 (+ 93us) service_pool.cc:225] Handling call
0112 02:57:25.350667 (+3236519us) ts_tablet_manager.cc:1145] Deleting tablet 3781431324bc479d97bd22b6bb1243b9
0112 02:57:25.350724 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.361210 (+ 10486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3219839}
I20250112 02:57:25.362823 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 119) took 3247 ms. Trace:
I20250112 02:57:25.363158 22379 rpcz_store.cc:276] 0112 02:57:22.114987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.115072 (+ 85us) service_pool.cc:225] Handling call
0112 02:57:25.362606 (+3247534us) ts_tablet_manager.cc:1145] Deleting tablet 1e7d4e98b6fa4aecb5955e8603eac779
0112 02:57:25.362644 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.362775 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3247070}
I20250112 02:57:25.364084 22379 tablet_replica.cc:331] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:25.364832 22379 raft_consensus.cc:2238] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:25.365947 22379 raft_consensus.cc:2267] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.366622 22457 consensus_queue.cc:1035] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:25.391459 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_661154f6ad374f1f809b070c6274bd4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:25.394820 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_661154f6ad374f1f809b070c6274bd4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.398722 22379 ts_tablet_manager.cc:1905] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.398662 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 3c4998c034e54b1bae0ef132cef5b04d
I20250112 02:57:25.402626 22579 raft_consensus.cc:491] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.403056 22579 raft_consensus.cc:513] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.405126 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e0b968d5b3cb4e34abd163429926d88e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.406010 22579 leader_election.cc:290] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:25.406193 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:25.406853 21751 raft_consensus.cc:2388] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
W20250112 02:57:25.406996 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0b968d5b3cb4e34abd163429926d88e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.411464 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:25.384661 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994e1d114e5a4de5aa1d7651907d718e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:25.415098 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21dc0755b38746a39bd773780e48cda5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:25.417785 21836 leader_election.cc:304] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:25.418834 22579 raft_consensus.cc:3054] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.420101 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_6eb2a5055484422a8c96499ae9e4f9db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:25.421862 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6eb2a5055484422a8c96499ae9e4f9db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.437944 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.444577 22375 tablet_replica.cc:331] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:25.445174 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 99e81199f79d46d6922eb1c6805d25da (table default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]) successfully deleted
I20250112 02:57:25.446126 22145 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:25.446244 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5f787ffe65e34e8f9852d149fff9f947 because the tablet was not found. No further retry: Not found: Tablet not found: 5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:25.446537 22145 ts_tablet_manager.cc:1397] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.260s user 0.003s sys 0.007s
W20250112 02:57:25.447307 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet f9bb62ab3f274c29b2c5f9528890b816 because the tablet was not found. No further retry: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816
W20250112 02:57:25.448294 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 33a5c293bbd643aab9409c6e9abe4f0a because the tablet was not found. No further retry: Not found: Tablet not found: 33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:25.454397 22145 raft_consensus.cc:357] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.455001 22145 raft_consensus.cc:383] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.455219 22145 raft_consensus.cc:738] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.457226 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_202c2f292aed4f11a310b5cf5c271aeb" } modify_external_catalogs: true
I20250112 02:57:25.457890 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_202c2f292aed4f11a310b5cf5c271aeb" } modify_external_catalogs: true
I20250112 02:57:25.463387 21690 leader_election.cc:304] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:25.464362 22645 raft_consensus.cc:3054] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.481173 21388 master_service.cc:946] ReplaceTablet: received request to replace tablet 98aa732d27f144aaad088ecc072837e9 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:25.496201 22379 ts_tablet_manager.cc:1918] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:25.496541 22379 log.cc:1198] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:25.497164 22379 ts_tablet_manager.cc:1939] T 7847b7af9e0b4aaba1f7cd06534f6510 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:25.498011 21752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa732d27f144aaad088ecc072837e9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:25.498935 21752 raft_consensus.cc:2463] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:25.504503 22143 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:25.505851 22143 ts_tablet_manager.cc:1397] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.244s user 0.040s sys 0.060s
I20250112 02:57:25.508458 22143 raft_consensus.cc:357] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.509104 22143 raft_consensus.cc:383] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.509425 22143 raft_consensus.cc:738] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.511952 22375 raft_consensus.cc:2238] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:57:25.512560 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 49318.2us
I20250112 02:57:25.513852 22145 consensus_queue.cc:260] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.515969 21390 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ae7a8479142d4028817982eb1bdf10c2" } new_table_name: "default.Table_33ea8d4190104eb8837824b516dd2314" modify_external_catalogs: true
I20250112 02:57:25.514686 22143 consensus_queue.cc:260] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.518565 21690 leader_election.cc:304] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.463810 22689 tablet.cc:1721] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:25.524240 22375 raft_consensus.cc:2267] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:25.522962 21356 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 49029.2us
I20250112 02:57:25.526860 22684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:53076 (request call id 159) took 1140 ms. Trace:
I20250112 02:57:25.527833 22645 raft_consensus.cc:2743] T 557de3de9a4d4d0bac1dc16f66ab6b01 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:25.532146 22703 tablet.cc:1721] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:25.534073 22145 ts_tablet_manager.cc:1428] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.087s user 0.005s sys 0.000s
I20250112 02:57:25.539269 22145 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:25.545722 22524 consensus_queue.cc:1035] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:25.508354 22144 ts_tablet_manager.cc:1428] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.396s user 0.005s sys 0.001s
I20250112 02:57:25.449770 22579 raft_consensus.cc:2743] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:25.554781 22645 raft_consensus.cc:2798] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:25.555243 22645 raft_consensus.cc:695] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:25.555342 22144 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:25.555985 22645 consensus_queue.cc:237] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.491379 22702 tablet.cc:1721] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:25.566259 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 47830.5us
I20250112 02:57:25.567068 21690 leader_election.cc:304] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.567867 22672 raft_consensus.cc:2798] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:25.571961 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 5f787ffe65e34e8f9852d149fff9f947 (table default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea [id=e084f29dae5a41899d7cb805c1ea63b6]) successfully deleted
I20250112 02:57:25.571947 22672 raft_consensus.cc:491] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:25.572299 22672 raft_consensus.cc:3054] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:25.572540 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 11fbebff35c344bdb305ea1a6dbbefe4 because the tablet was not found. No further retry: Not found: Tablet not found: 11fbebff35c344bdb305ea1a6dbbefe4
W20250112 02:57:25.572393 21978 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.569403 22143 ts_tablet_manager.cc:1428] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.063s user 0.003s sys 0.003s
I20250112 02:57:25.572901 21978 leader_election.cc:304] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.520725 22704 tablet.cc:1721] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:25.571261 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:25.571563 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:25.573588 22143 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:25.577808 22375 ts_tablet_manager.cc:1905] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.527163 22684 rpcz_store.cc:276] 0112 02:57:24.386062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.386805 (+ 743us) service_pool.cc:225] Handling call
0112 02:57:25.526801 (+1139996us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:24.387617 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0112 02:57:24.437959 (+ 50342us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0112 02:57:24.437988 (+ 29us) alter_schema_op.cc:66] Acquired schema lock
0112 02:57:24.438028 (+ 40us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0112 02:57:24.438150 (+ 122us) alter_schema_op.cc:128] START. Timestamp: 1736650644438090
0112 02:57:24.438167 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:24.438871 (+ 704us) log.cc:844] Serialized 173 byte log entry
0112 02:57:25.191272 (+752401us) op_driver.cc:464] REPLICATION: finished
0112 02:57:25.254680 (+ 63408us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0112 02:57:25.303667 (+ 48987us) log.cc:844] Serialized 5 byte log entry
0112 02:57:25.489477 (+185810us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.525602 (+ 36125us) log.cc:844] Serialized 14 byte log entry
0112 02:57:25.526093 (+ 491us) alter_schema_op.cc:73] Released schema lock
0112 02:57:25.526124 (+ 31us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63196,"mutex_wait_us":5538,"prepare.queue_time_us":128,"prepare.run_cpu_time_us":44363,"prepare.run_wall_time_us":638220,"raft.queue_time_us":91621,"raft.run_cpu_time_us":1878,"raft.run_wall_time_us":126986,"replication_time_us":753014,"thread_start_us":608155,"threads_started":4,"wal-append.queue_time_us":213316}]]}
I20250112 02:57:25.575038 21978 leader_election.cc:304] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:25.573894 22581 raft_consensus.cc:2743] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:25.592473 22457 raft_consensus.cc:2798] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:25.574126 22652 raft_consensus.cc:491] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:25.592826 22457 raft_consensus.cc:491] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:25.593166 22457 raft_consensus.cc:3054] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:25.593104 22652 raft_consensus.cc:513] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.596469 22652 leader_election.cc:290] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:25.600349 22457 raft_consensus.cc:513] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:25.603847 21978 leader_election.cc:343] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: c48d1bc147ba48ebbebfd707dc44fe02
W20250112 02:57:25.605314 21978 consensus_peers.cc:487] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33a5c293bbd643aab9409c6e9abe4f0a. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:25.606904 22376 ts_tablet_manager.cc:1918] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:25.607240 22376 log.cc:1198] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:25.607851 22376 ts_tablet_manager.cc:1939] T 4166dcd1f9d74255b21810dd42b7358d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:25.609165 22672 raft_consensus.cc:513] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:25.603698 22457 leader_election.cc:290] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:25.613574 22143 tablet_bootstrap.cc:654] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.628091 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa732d27f144aaad088ecc072837e9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:25.631119 22672 leader_election.cc:290] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:25.635473 21980 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.636283 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:25.639710 21980 leader_election.cc:343] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:25.644531 22144 tablet_bootstrap.cc:654] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.650449 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_06528f9a9c8d4cb991d30e195f3944be" } new_table_name: "default.Table_eec91efaefdb4291a65cc6808710cae3" modify_external_catalogs: true
I20250112 02:57:25.653901 21387 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e589965a6e364011b9d11bea87be94c2" } modify_external_catalogs: true
I20250112 02:57:25.654389 21387 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e589965a6e364011b9d11bea87be94c2" } modify_external_catalogs: true
I20250112 02:57:25.656929 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 113) took 4193 ms. Trace:
I20250112 02:57:25.657263 22376 rpcz_store.cc:276] 0112 02:57:21.463603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.477206 (+ 13603us) service_pool.cc:225] Handling call
0112 02:57:25.161118 (+3683912us) ts_tablet_manager.cc:1145] Deleting tablet 4166dcd1f9d74255b21810dd42b7358d
0112 02:57:25.161158 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.456982 (+295824us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.606683 (+149701us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.656869 (+ 50186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89640,"tablet-delete.queue_time_us":3557762}
I20250112 02:57:25.657987 21389 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3f6b3b37133b44cf9918667e59315645"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.658306 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 4166dcd1f9d74255b21810dd42b7358d (table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]) successfully deleted
I20250112 02:57:25.658638 22376 tablet_replica.cc:331] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:25.659354 22376 raft_consensus.cc:2238] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:25.659366 21389 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f6b3b37133b44cf9918667e59315645 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.659958 22376 pending_rounds.cc:62] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Trying to abort 1 pending ops.
I20250112 02:57:25.660192 22376 pending_rounds.cc:69] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321043333058560 op_type: NO_OP noop_request { }
I20250112 02:57:25.660559 22376 raft_consensus.cc:2883] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:25.660866 22376 raft_consensus.cc:2267] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.663209 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 120) took 3547 ms. Trace:
I20250112 02:57:25.663770 22379 rpcz_store.cc:276] 0112 02:57:22.115808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.115904 (+ 96us) service_pool.cc:225] Handling call
0112 02:57:25.363959 (+3248055us) ts_tablet_manager.cc:1145] Deleting tablet 7847b7af9e0b4aaba1f7cd06534f6510
0112 02:57:25.363993 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.451407 (+ 87414us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.496008 (+ 44601us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.663110 (+167102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3247622}
I20250112 02:57:25.665530 22379 tablet_replica.cc:331] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:25.692212 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 4716677cfad74c0dafe18c55796aeae5 (DEFAULT_TABLE table=default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:25.694440 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:25.697196 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 4716677cfad74c0dafe18c55796aeae5 (DEFAULT_TABLE table=default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:25.698410 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4716677cfad74c0dafe18c55796aeae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:25.703576 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:25.703855 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4716677cfad74c0dafe18c55796aeae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:25.704540 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fe67a814504a89b6fa8f46e2a4aee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:25.706975 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fe67a814504a89b6fa8f46e2a4aee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:25.707000 21385 catalog_manager.cc:5526] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.709585 21385 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 3 got 0
W20250112 02:57:25.720433 21390 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2601daad8ba742ec88ccfe603440e8e7 consensus state has no leader
I20250112 02:57:25.710685 21385 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:25.731534 21385 catalog_manager.cc:5526] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.736343 21385 catalog_manager.cc:5526] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.745710 22375 ts_tablet_manager.cc:1918] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:57:25.745131 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a consensus state has no leader
I20250112 02:57:25.754648 22375 log.cc:1198] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/afeff74abcc643a5ad6565d9dcc40d62
I20250112 02:57:25.755424 22375 ts_tablet_manager.cc:1939] T afeff74abcc643a5ad6565d9dcc40d62 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:25.667758 22143 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:25.758270 22143 ts_tablet_manager.cc:1397] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.185s user 0.008s sys 0.004s
I20250112 02:57:25.760478 22143 raft_consensus.cc:357] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.761128 22143 raft_consensus.cc:383] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.761361 22143 raft_consensus.cc:738] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.761865 22143 consensus_queue.cc:260] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.763886 22143 ts_tablet_manager.cc:1428] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250112 02:57:25.764613 22143 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:25.767390 21387 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet af2c0816a4e54b258dd3e87c09f1e381
I20250112 02:57:25.769119 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet afeff74abcc643a5ad6565d9dcc40d62 (table default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]) successfully deleted
I20250112 02:57:25.772900 22143 tablet_bootstrap.cc:654] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.753607 22144 tablet_bootstrap.cc:492] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
W20250112 02:57:25.772930 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 301c8a2c534a4c54bfd6e678f3184ea4 because the tablet was not found. No further retry: Not found: Tablet not found: 301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:25.773309 22144 ts_tablet_manager.cc:1397] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.221s user 0.008s sys 0.004s
W20250112 02:57:25.774381 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 7d699356dd3341338670b433a7fa02ad because the tablet was not found. No further retry: Not found: Tablet not found: 7d699356dd3341338670b433a7fa02ad
I20250112 02:57:25.775596 22144 raft_consensus.cc:357] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.776592 22375 tablet_replica.cc:331] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:25.778420 22375 raft_consensus.cc:2238] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:25.780026 22145 tablet_bootstrap.cc:654] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.783161 22375 raft_consensus.cc:2267] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.786578 22379 raft_consensus.cc:2238] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:25.787149 22379 raft_consensus.cc:2267] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.787365 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 443aa544f6df4d50a78c9effd306e216
I20250112 02:57:25.787933 22375 ts_tablet_manager.cc:1905] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.788575 21383 catalog_manager.cc:6596] ReplaceTablet: tablet 443aa544f6df4d50a78c9effd306e216 deleted and replaced by tablet b9034065c6bc435d9f26f85a96297cd3
I20250112 02:57:25.789427 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 443aa544f6df4d50a78c9effd306e216 with delete_type TABLET_DATA_DELETED (replaced by tablet b9034065c6bc435d9f26f85a96297cd3) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:25.789497 22376 ts_tablet_manager.cc:1905] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.790616 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 443aa544f6df4d50a78c9effd306e216 with delete_type TABLET_DATA_DELETED (replaced by tablet b9034065c6bc435d9f26f85a96297cd3) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.777509 22524 consensus_queue.cc:1035] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250112 02:57:25.796461 21386 catalog_manager.cc:5526] T 7af7e02e5fc248839b434f42f95c6509 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.777304 22144 raft_consensus.cc:383] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.801899 22144 raft_consensus.cc:738] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.804160 22144 consensus_queue.cc:260] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.805101 22379 ts_tablet_manager.cc:1905] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.810950 22145 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:25.811442 22145 ts_tablet_manager.cc:1397] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.272s user 0.009s sys 0.002s
I20250112 02:57:25.813546 22145 raft_consensus.cc:357] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.797840 21386 catalog_manager.cc:5526] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.814126 22145 raft_consensus.cc:383] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.814940 21386 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 3 got 0
I20250112 02:57:25.816256 21386 catalog_manager.cc:5526] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:25.821462 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:25.823606 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.824312 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_49dc0df59ea941a5a47e56ae5218d8f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.825584 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:25.828254 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49dc0df59ea941a5a47e56ae5218d8f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.829828 22375 ts_tablet_manager.cc:1918] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:25.830247 22375 log.cc:1198] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:25.830659 22144 ts_tablet_manager.cc:1428] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.057s user 0.006s sys 0.001s
I20250112 02:57:25.831079 22375 ts_tablet_manager.cc:1939] T 7847b7af9e0b4aaba1f7cd06534f6510 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:25.835135 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_706a3ed7905244fbbf03303bdbde0fb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:25.836459 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_706a3ed7905244fbbf03303bdbde0fb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.839636 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 157) took 1028 ms. Trace:
I20250112 02:57:25.839982 22375 rpcz_store.cc:276] 0112 02:57:24.811016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.811127 (+ 111us) service_pool.cc:225] Handling call
0112 02:57:25.776413 (+965286us) ts_tablet_manager.cc:1145] Deleting tablet 7847b7af9e0b4aaba1f7cd06534f6510
0112 02:57:25.776460 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.803433 (+ 26973us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.829630 (+ 26197us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.839585 (+ 9955us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":964670}
I20250112 02:57:25.814348 22145 raft_consensus.cc:738] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.841465 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 7847b7af9e0b4aaba1f7cd06534f6510 (table default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]) successfully deleted
I20250112 02:57:25.841784 22145 consensus_queue.cc:260] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.843753 22145 ts_tablet_manager.cc:1428] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.032s user 0.004s sys 0.001s
I20250112 02:57:25.843873 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_02aed393b2e34f1db7553def2ae78759"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.831490 22144 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:25.855564 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02aed393b2e34f1db7553def2ae78759 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.858992 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 152) took 1053 ms. Trace:
I20250112 02:57:25.859372 22375 rpcz_store.cc:276] 0112 02:57:24.805373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.854493 (+ 49120us) service_pool.cc:225] Handling call
0112 02:57:25.858761 (+1004268us) ts_tablet_manager.cc:1145] Deleting tablet 4166dcd1f9d74255b21810dd42b7358d
0112 02:57:25.858800 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.858940 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003710}
W20250112 02:57:25.860628 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 4166dcd1f9d74255b21810dd42b7358d because the tablet was not found. No further retry: Not found: Tablet not found: 4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:25.865242 22144 tablet_bootstrap.cc:654] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.867146 22145 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:25.866403 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet f9bb62ab3f274c29b2c5f9528890b816 with delete_type TABLET_DATA_DELETED (replaced by tablet e8f7301bed6c4d98a80c12b70073e62f) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:25.876463 22145 tablet_bootstrap.cc:654] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.890242 22376 ts_tablet_manager.cc:1918] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250112 02:57:25.894927 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f98f6840e24f3c9d03937c1bbc00e7 already in progress: creating tablet
I20250112 02:57:25.895411 22376 log.cc:1198] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/c48d1bc147ba48ebbebfd707dc44fe02
W20250112 02:57:25.896664 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 98aa732d27f144aaad088ecc072837e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98aa732d27f144aaad088ecc072837e9 already in progress: creating tablet
I20250112 02:57:25.897327 22376 ts_tablet_manager.cc:1939] T c48d1bc147ba48ebbebfd707dc44fe02 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:25.898101 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5e3890251e644c1eb596de043b63491e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e3890251e644c1eb596de043b63491e already in progress: creating tablet
I20250112 02:57:25.908705 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 118) took 4065 ms. Trace:
W20250112 02:57:25.914155 21385 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 29) took 881 ms (client timeout 1000 ms). Trace:
I20250112 02:57:25.915378 22376 rpcz_store.cc:276] 0112 02:57:21.843570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.843682 (+ 112us) service_pool.cc:225] Handling call
0112 02:57:25.658474 (+3814792us) ts_tablet_manager.cc:1145] Deleting tablet c48d1bc147ba48ebbebfd707dc44fe02
0112 02:57:25.658531 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.825413 (+166882us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.890018 (+ 64605us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:25.908650 (+ 18632us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3814299}
I20250112 02:57:25.917572 22376 tablet_replica.cc:331] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:25.917055 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet f445a4bf43db4571b6d3a4afe8fe823a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f445a4bf43db4571b6d3a4afe8fe823a already in progress: creating tablet
I20250112 02:57:25.918902 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet c48d1bc147ba48ebbebfd707dc44fe02 (table default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]) successfully deleted
I20250112 02:57:25.921571 22144 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:25.921983 22144 ts_tablet_manager.cc:1397] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.002s
I20250112 02:57:25.923890 22375 tablet_replica.cc:331] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:25.925290 22376 raft_consensus.cc:2238] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:25.924098 22144 raft_consensus.cc:357] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.926398 22144 raft_consensus.cc:383] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.926676 22144 raft_consensus.cc:738] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.927245 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_267a17c8b93540128cfb0744673fe49e" } new_table_name: "default.Table_2b8baa60a7364dc0b97fac05ac195d94" modify_external_catalogs: true
I20250112 02:57:25.927249 22144 consensus_queue.cc:260] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.928098 22145 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:25.929684 22144 ts_tablet_manager.cc:1428] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250112 02:57:25.928475 22145 ts_tablet_manager.cc:1397] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.004s
I20250112 02:57:25.931005 22144 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:25.934484 22145 raft_consensus.cc:357] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.935093 22145 raft_consensus.cc:383] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:25.935326 22145 raft_consensus.cc:738] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:25.936192 22376 raft_consensus.cc:2267] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.936671 22144 tablet_bootstrap.cc:654] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:25.941262 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_256609c2ce914c68ad16988dc5dd7ca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:25.943079 22375 raft_consensus.cc:2238] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:57:25.943032 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_256609c2ce914c68ad16988dc5dd7ca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:25.943575 22375 raft_consensus.cc:2267] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:25.943783 22376 ts_tablet_manager.cc:1905] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:25.945173 22375 ts_tablet_manager.cc:1905] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:25.920433 21385 rpcz_store.cc:269] 0112 02:57:25.033174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.033290 (+ 116us) service_pool.cc:225] Handling call
0112 02:57:25.914080 (+880790us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:25.759560 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:25.759840 (+ 280us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:25.759859 (+ 19us) write_op.cc:435] Acquired schema lock
0112 02:57:25.759870 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:25.760272 (+ 402us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:25.760303 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:25.760323 (+ 20us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:25.760546 (+ 223us) tablet.cc:698] Row locks acquired
0112 02:57:25.760560 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:25.760653 (+ 93us) write_op.cc:270] Start()
0112 02:57:25.760725 (+ 72us) write_op.cc:276] Timestamp: P: 1736650645760724 usec, L: 0
0112 02:57:25.760738 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:25.761135 (+ 397us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:25.779319 (+ 18184us) op_driver.cc:464] REPLICATION: finished
0112 02:57:25.779675 (+ 356us) write_op.cc:301] APPLY: starting
0112 02:57:25.779726 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:25.779837 (+ 111us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:25.779960 (+ 123us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:25.783674 (+ 3714us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:25.783814 (+ 140us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:25.783853 (+ 39us) write_op.cc:312] APPLY: finished
0112 02:57:25.784031 (+ 178us) log.cc:844] Serialized 35 byte log entry
0112 02:57:25.784898 (+ 867us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:25.784962 (+ 64us) write_op.cc:454] Released schema lock
0112 02:57:25.784991 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:25.785189 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:25.785212 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":142663,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":243,"apply.run_cpu_time_us":4037,"apply.run_wall_time_us":5478,"num_ops":3,"prepare.queue_time_us":6456,"prepare.run_cpu_time_us":1823,"prepare.run_wall_time_us":2826,"raft.queue_time_us":65,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":18519}]]}
I20250112 02:57:25.953192 21387 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_50bcfd9aa201416e976e67c56aa7f3f6" } new_table_name: "default.Table_864627e87b84488693c05eabec65701d" modify_external_catalogs: true
I20250112 02:57:25.945636 22145 consensus_queue.cc:260] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:25.963125 22145 ts_tablet_manager.cc:1428] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20250112 02:57:25.963903 22145 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:25.965000 22689 tablet.cc:1721] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:25.968205 22703 tablet.cc:1721] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:25.981429 22145 tablet_bootstrap.cc:654] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:26.010212 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 443181us
I20250112 02:57:26.013051 22143 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:26.014127 22143 ts_tablet_manager.cc:1397] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.250s user 0.011s sys 0.005s
I20250112 02:57:26.015784 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 7847b7af9e0b4aaba1f7cd06534f6510 (table default.Table_1ab713178ff6472aa38bb55cd1c55d5e [id=07d4c9ab92a243829cca839af03af0fe]) successfully deleted
I20250112 02:57:26.016175 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9543a0a0359e4b72bd30f1cb6290f767" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:26.018447 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9543a0a0359e4b72bd30f1cb6290f767" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:26.019004 22044 raft_consensus.cc:3054] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.018796 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:26.021116 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa732d27f144aaad088ecc072837e9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:26.021868 21753 raft_consensus.cc:3054] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.023159 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_102d4e698b854093aa5caa6c1e25ad50" } modify_external_catalogs: true
I20250112 02:57:26.022990 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 443aa544f6df4d50a78c9effd306e216 with delete_type TABLET_DATA_DELETED (replaced by tablet b9034065c6bc435d9f26f85a96297cd3) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.023644 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_102d4e698b854093aa5caa6c1e25ad50" } modify_external_catalogs: true
I20250112 02:57:26.024335 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:26.033929 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 34) took 1060 ms (client timeout 1000 ms). Trace:
I20250112 02:57:26.026288 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 4716677cfad74c0dafe18c55796aeae5 (DEFAULT_TABLE table=default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:26.035485 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4716677cfad74c0dafe18c55796aeae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:26.042958 22044 raft_consensus.cc:2463] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:26.052927 22581 raft_consensus.cc:491] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.053436 22581 raft_consensus.cc:513] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.055177 22581 leader_election.cc:290] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:26.058775 22145 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.059204 22145 ts_tablet_manager.cc:1397] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.003s
I20250112 02:57:26.059423 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_3dfd9d3697264cfc9249b9f772b58181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.058758 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:26.061591 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dfd9d3697264cfc9249b9f772b58181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:26.061559 21980 leader_election.cc:343] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.061626 22145 raft_consensus.cc:357] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.062252 22145 raft_consensus.cc:383] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.080901 22724 raft_consensus.cc:491] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.082798 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9
I20250112 02:57:26.084597 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:26.085060 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:26.034261 21386 rpcz_store.cc:269] 0112 02:57:24.973594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.975466 (+ 1872us) service_pool.cc:225] Handling call
0112 02:57:26.033867 (+1058401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:25.897851 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:25.898519 (+ 668us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:25.898541 (+ 22us) write_op.cc:435] Acquired schema lock
0112 02:57:25.898552 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:25.898947 (+ 395us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:25.898984 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:25.899007 (+ 23us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:25.899234 (+ 227us) tablet.cc:698] Row locks acquired
0112 02:57:25.899247 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:25.899357 (+ 110us) write_op.cc:270] Start()
0112 02:57:25.899428 (+ 71us) write_op.cc:276] Timestamp: P: 1736650645899475 usec, L: 0
0112 02:57:25.899442 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:25.925591 (+ 26149us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b5400042098. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eee3650f4 00007f7eed605389 00007f7eed5c11f1 00007f7eed5c0eca 00007f7eed62ccd5 00007f7eed62d179 00007f7eeda993b1 00007f7eeda98b88 00007f7eeda9d821 00007f7eeda9d7d9 00007f7eeda9d769
0112 02:57:25.925829 (+ 238us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:26.026722 (+100893us) op_driver.cc:464] REPLICATION: finished
0112 02:57:26.028273 (+ 1551us) write_op.cc:301] APPLY: starting
0112 02:57:26.028329 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:26.028443 (+ 114us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:26.028462 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:26.028739 (+ 277us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:26.028876 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:26.028912 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:26.029085 (+ 173us) log.cc:844] Serialized 35 byte log entry
0112 02:57:26.029217 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:26.029287 (+ 70us) write_op.cc:454] Released schema lock
0112 02:57:26.029313 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:26.029514 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:26.029537 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":1398,"apply.run_cpu_time_us":1221,"apply.run_wall_time_us":1220,"mutex_wait_us":33560,"num_ops":3,"prepare.queue_time_us":33673,"prepare.run_cpu_time_us":3608,"prepare.run_wall_time_us":61623,"raft.queue_time_us":32515,"raft.run_cpu_time_us":740,"raft.run_wall_time_us":742,"replication_time_us":127200,"spinlock_wait_cycles":52919552}]]}
I20250112 02:57:26.084148 21735 tablet_service.cc:1467] Processing CreateTablet for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (DEFAULT_TABLE table=default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:26.108927 22375 ts_tablet_manager.cc:1918] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.109259 22375 log.cc:1198] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/0bca1c68aae24a1f9974a0bd2244fbf8
I20250112 02:57:26.109923 22375 ts_tablet_manager.cc:1939] T 0bca1c68aae24a1f9974a0bd2244fbf8 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:26.099625 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e422009a82534de8919b09f667f83a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.111407 22145 raft_consensus.cc:738] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:26.112007 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e422009a82534de8919b09f667f83a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.112208 22376 ts_tablet_manager.cc:1918] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.112566 22376 log.cc:1198] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:26.114877 22145 consensus_queue.cc:260] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.116601 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66fe67a814504a89b6fa8f46e2a4aee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:26.103286 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_8e51a35f779844a3a2e5fec4500f7697"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.118156 22376 ts_tablet_manager.cc:1939] T 33a5c293bbd643aab9409c6e9abe4f0a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:26.104957 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cf8bf172abad43ac9a519310334d1640 consensus state has no leader
I20250112 02:57:26.107985 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3f6b3b37133b44cf9918667e59315645" } modify_external_catalogs: true
I20250112 02:57:26.119848 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3f6b3b37133b44cf9918667e59315645" } modify_external_catalogs: true
I20250112 02:57:26.107304 21384 catalog_manager.cc:5526] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
W20250112 02:57:26.122937 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e51a35f779844a3a2e5fec4500f7697 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.123948 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]). Expected version 3 got 2
I20250112 02:57:26.125018 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]). Expected version 1 got 0
I20250112 02:57:26.126103 22145 ts_tablet_manager.cc:1428] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20250112 02:57:26.128302 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 120) took 4203 ms. Trace:
I20250112 02:57:26.128604 22376 rpcz_store.cc:276] 0112 02:57:21.925309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.925467 (+ 158us) service_pool.cc:225] Handling call
0112 02:57:25.917430 (+3991963us) ts_tablet_manager.cc:1145] Deleting tablet 33a5c293bbd643aab9409c6e9abe4f0a
0112 02:57:25.917472 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:25.985624 (+ 68152us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.110771 (+125147us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.128249 (+ 17478us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8281,"tablet-delete.queue_time_us":3991297}
I20250112 02:57:26.135435 22145 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.140357 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 33a5c293bbd643aab9409c6e9abe4f0a (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]) successfully deleted
I20250112 02:57:26.145318 22145 tablet_bootstrap.cc:654] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.134325 22724 raft_consensus.cc:513] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.158344 22644 raft_consensus.cc:491] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.158728 22724 leader_election.cc:290] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.164300 22672 raft_consensus.cc:491] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cca160b7cffb4ae287ea077e7352ffc9)
I20250112 02:57:26.164867 22672 raft_consensus.cc:513] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.145996 22376 tablet_replica.cc:331] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.179491 22376 raft_consensus.cc:2238] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:26.180341 22376 raft_consensus.cc:2267] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.182065 22376 ts_tablet_manager.cc:1905] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.187057 22375 tablet_replica.cc:331] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:26.188884 22375 raft_consensus.cc:2238] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:26.191390 21384 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 3 got 0
I20250112 02:57:26.193302 22144 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:26.194821 22144 ts_tablet_manager.cc:1397] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.264s user 0.016s sys 0.001s
I20250112 02:57:26.194805 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5]) successfully deleted
W20250112 02:57:26.195441 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 9543a0a0359e4b72bd30f1cb6290f767 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9543a0a0359e4b72bd30f1cb6290f767 already in progress: creating tablet
I20250112 02:57:26.198098 22144 raft_consensus.cc:357] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.199077 22144 raft_consensus.cc:383] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.199450 22144 raft_consensus.cc:738] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.200237 22144 consensus_queue.cc:260] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.203404 22144 ts_tablet_manager.cc:1428] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250112 02:57:26.207671 21385 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 799b2bc2a33247f7a223e95213a0aab2
W20250112 02:57:26.208854 21978 leader_election.cc:343] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.209267 21978 leader_election.cc:304] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.158874 22644 raft_consensus.cc:513] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.211236 22581 raft_consensus.cc:2743] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:26.211628 22375 raft_consensus.cc:2267] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.212229 22644 leader_election.cc:290] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.212745 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:26.218372 22672 leader_election.cc:290] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.227756 22145 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.228096 22145 ts_tablet_manager.cc:1397] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.005s
I20250112 02:57:26.229782 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:26.230053 22145 raft_consensus.cc:357] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.231097 22145 raft_consensus.cc:383] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.231334 22145 raft_consensus.cc:738] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.231873 22145 consensus_queue.cc:260] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.234261 22144 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:26.236059 22145 ts_tablet_manager.cc:1428] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250112 02:57:26.237112 22145 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.242092 22145 tablet_bootstrap.cc:654] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.251426 21753 raft_consensus.cc:2463] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:26.252286 22144 tablet_bootstrap.cc:654] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.252936 22143 raft_consensus.cc:357] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.253722 22143 raft_consensus.cc:383] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.254029 22143 raft_consensus.cc:738] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.254766 22143 consensus_queue.cc:260] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.266464 22376 ts_tablet_manager.cc:1918] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.266994 22376 log.cc:1198] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:26.267824 22376 ts_tablet_manager.cc:1939] T 5f787ffe65e34e8f9852d149fff9f947 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:26.270162 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 122) took 4294 ms. Trace:
I20250112 02:57:26.287781 22585 consensus_queue.cc:1035] T b4dddfea8e8b49189186930fce46ecc5 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:26.292505 22376 rpcz_store.cc:276] 0112 02:57:21.976097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:21.976602 (+ 505us) service_pool.cc:225] Handling call
0112 02:57:26.145845 (+4169243us) ts_tablet_manager.cc:1145] Deleting tablet 5f787ffe65e34e8f9852d149fff9f947
0112 02:57:26.145883 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.209417 (+ 63534us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.266217 (+ 56800us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.270088 (+ 3871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4168707}
I20250112 02:57:26.295513 22379 ts_tablet_manager.cc:1918] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.301824 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 127) took 4225 ms. Trace:
W20250112 02:57:26.218082 21387 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 consensus state has no leader
I20250112 02:57:26.309222 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_933de7797e444faa9a9c4315ba2fe282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.302156 22376 rpcz_store.cc:276] 0112 02:57:22.075867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.078337 (+ 2470us) service_pool.cc:225] Handling call
0112 02:57:26.293710 (+4215373us) ts_tablet_manager.cc:1145] Deleting tablet 7847b7af9e0b4aaba1f7cd06534f6510
0112 02:57:26.301574 (+ 7864us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.301746 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214633}
W20250112 02:57:26.310653 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_933de7797e444faa9a9c4315ba2fe282 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.311053 22376 tablet_replica.cc:331] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.287597 21388 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2974589a1bbd43d0b58eac1aeed1c19f" } modify_external_catalogs: true
I20250112 02:57:26.293809 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 5f787ffe65e34e8f9852d149fff9f947 (table default.Table_b7afbaa60aaf4bdabaf0f94bc5f647ea [id=e084f29dae5a41899d7cb805c1ea63b6]) successfully deleted
I20250112 02:57:26.311436 21388 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2974589a1bbd43d0b58eac1aeed1c19f" } modify_external_catalogs: true
W20250112 02:57:26.311684 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 7847b7af9e0b4aaba1f7cd06534f6510 because the tablet was not found. No further retry: Not found: Tablet not found: 7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:26.303421 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e429d768998e4607aff0d49ffecbc300"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.312003 22376 raft_consensus.cc:2238] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:26.312880 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e429d768998e4607aff0d49ffecbc300 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.317469 22376 raft_consensus.cc:2267] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.307797 22375 ts_tablet_manager.cc:1905] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.321455 21385 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_eec91efaefdb4291a65cc6808710cae3" } new_table_name: "default.Table_948d7e0db47d4a47b2d6e6ec21baf292" modify_external_catalogs: true
I20250112 02:57:26.302218 22644 raft_consensus.cc:491] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.328670 22379 log.cc:1198] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/7d699356dd3341338670b433a7fa02ad
I20250112 02:57:26.328948 22644 raft_consensus.cc:513] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:26.302860 21691 leader_election.cc:343] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:26.330873 21691 leader_election.cc:343] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:26.331691 21691 consensus_peers.cc:487] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afeff74abcc643a5ad6565d9dcc40d62. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:26.332172 21691 leader_election.cc:343] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Not found: Tablet not found: 301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:26.305085 22143 ts_tablet_manager.cc:1428] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.290s user 0.002s sys 0.004s
I20250112 02:57:26.316262 22728 consensus_queue.cc:1035] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:26.335609 22143 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:26.336763 22376 ts_tablet_manager.cc:1905] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.337812 22728 consensus_queue.cc:1035] T eef30508946e4376961b6cd463172c1f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:26.338744 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:26.339903 21978 leader_election.cc:343] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.340327 21978 leader_election.cc:304] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.341394 22379 ts_tablet_manager.cc:1939] T 7d699356dd3341338670b433a7fa02ad P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:26.341364 21978 leader_election.cc:304] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.341971 22581 raft_consensus.cc:2743] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:26.342147 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 129) took 3592 ms. Trace:
I20250112 02:57:26.342478 22730 raft_consensus.cc:2798] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.342919 22730 raft_consensus.cc:695] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:26.343600 22730 consensus_queue.cc:237] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.348613 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e7b514137af3470790430ad30dcb6c5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.348851 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_69f02882e385448497a47ae9c2398e7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.349628 22644 leader_election.cc:290] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:26.350263 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69f02882e385448497a47ae9c2398e7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.350832 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:26.350260 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7b514137af3470790430ad30dcb6c5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:26.353768 21691 leader_election.cc:343] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.359863 22143 tablet_bootstrap.cc:654] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.366765 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:26.371685 21691 leader_election.cc:343] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.342422 22379 rpcz_store.cc:276] 0112 02:57:22.750003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.751194 (+ 1191us) service_pool.cc:225] Handling call
0112 02:57:25.665151 (+2913957us) ts_tablet_manager.cc:1145] Deleting tablet 7d699356dd3341338670b433a7fa02ad
0112 02:57:25.665224 (+ 73us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.246901 (+581677us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.295210 (+ 48309us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.342084 (+ 46874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2913336}
I20250112 02:57:26.379985 22379 tablet_replica.cc:331] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:26.380748 22379 raft_consensus.cc:2238] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.381206 22379 raft_consensus.cc:2267] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.382426 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]) successfully deleted
I20250112 02:57:26.384572 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.391194 22379 ts_tablet_manager.cc:1905] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:26.401669 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_eec91efaefdb4291a65cc6808710cae3 [id=2fa952f6cb4e4fd5a40b77000c30d75e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a consensus state has no leader
I20250112 02:57:26.405872 22379 ts_tablet_manager.cc:1918] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:26.406308 22379 log.cc:1198] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:26.408142 22379 ts_tablet_manager.cc:1939] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:26.408867 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 130) took 3137 ms. Trace:
I20250112 02:57:26.409179 22379 rpcz_store.cc:276] 0112 02:57:23.271033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.271154 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:26.379816 (+3108662us) ts_tablet_manager.cc:1145] Deleting tablet 301c8a2c534a4c54bfd6e678f3184ea4
0112 02:57:26.379888 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.398402 (+ 18514us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.405683 (+ 7281us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.408818 (+ 3135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108049}
I20250112 02:57:26.410388 22379 tablet_replica.cc:331] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:26.411767 22379 raft_consensus.cc:2238] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.412285 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 301c8a2c534a4c54bfd6e678f3184ea4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]) successfully deleted
I20250112 02:57:26.412572 22379 raft_consensus.cc:2267] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.418821 22379 ts_tablet_manager.cc:1905] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.433698 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 301c8a2c534a4c54bfd6e678f3184ea4 with delete_type TABLET_DATA_DELETED (replaced by tablet b0d0ba05ee4c4c9f815a4c04830b5dd4) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.433120 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.437848 21690 leader_election.cc:304] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.438385 22376 ts_tablet_manager.cc:1918] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.438755 22376 log.cc:1198] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/7d699356dd3341338670b433a7fa02ad
I20250112 02:57:26.440348 22379 ts_tablet_manager.cc:1918] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:26.439903 22143 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:26.440670 22379 log.cc:1198] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:26.440755 22644 raft_consensus.cc:2798] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.441946 22379 ts_tablet_manager.cc:1939] T 77d80c6a60bc46449c7c0656128100c8 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:26.442469 22644 raft_consensus.cc:695] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:26.443630 22644 consensus_queue.cc:237] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.441104 22143 ts_tablet_manager.cc:1397] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.106s user 0.011s sys 0.001s
W20250112 02:57:26.440543 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 131025us
I20250112 02:57:26.448742 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.449916 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.450234 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.450819 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "301c8a2c534a4c54bfd6e678f3184ea4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.451855 22043 raft_consensus.cc:2463] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:26.456218 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 138) took 2643 ms. Trace:
I20250112 02:57:26.456537 22379 rpcz_store.cc:276] 0112 02:57:23.812586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.822114 (+ 9528us) service_pool.cc:225] Handling call
0112 02:57:26.410233 (+2588119us) ts_tablet_manager.cc:1145] Deleting tablet 77d80c6a60bc46449c7c0656128100c8
0112 02:57:26.410286 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.428675 (+ 18389us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.440158 (+ 11483us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.456163 (+ 16005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587634}
I20250112 02:57:26.458289 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]) successfully deleted
I20250112 02:57:26.458796 22376 ts_tablet_manager.cc:1939] T 7d699356dd3341338670b433a7fa02ad P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:26.458815 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 6fbe0e20a157479e9160fc6782e3bf37 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:26.460619 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 141) took 3710 ms. Trace:
I20250112 02:57:26.460924 22376 rpcz_store.cc:276] 0112 02:57:22.749917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:22.750069 (+ 152us) service_pool.cc:225] Handling call
0112 02:57:26.310891 (+3560822us) ts_tablet_manager.cc:1145] Deleting tablet 7d699356dd3341338670b433a7fa02ad
0112 02:57:26.310952 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.389159 (+ 78207us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.438175 (+ 49016us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.460526 (+ 22351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556078}
I20250112 02:57:26.461544 22143 raft_consensus.cc:357] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.462411 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 7d699356dd3341338670b433a7fa02ad (table default.Table_f2f1590c7a40439c89ff74db3d19cdcf [id=8aa9ef226db340358d3a91700aa33602]) successfully deleted
W20250112 02:57:26.468652 21384 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 37) took 1186 ms (client timeout 1000 ms). Trace:
I20250112 02:57:26.468744 21735 tablet_service.cc:1514] Processing DeleteTablet for tablet 7847b7af9e0b4aaba1f7cd06534f6510 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:18 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.469621 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d80c6a60bc46449c7c0656128100c8 with delete_type TABLET_DATA_DELETED (replaced by tablet 66fe67a814504a89b6fa8f46e2a4aee9) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.467871 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.480304 21386 master_service.cc:946] ReplaceTablet: received request to replace tablet 84f7700b8cce4261a32f2009d1481b99 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:26.462353 22143 raft_consensus.cc:383] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.484694 22143 raft_consensus.cc:738] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.487102 22145 tablet_bootstrap.cc:492] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.487661 22145 ts_tablet_manager.cc:1397] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.251s user 0.036s sys 0.044s
W20250112 02:57:26.488137 21690 leader_election.cc:343] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.488524 21690 leader_election.cc:304] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:26.489264 22143 consensus_queue.cc:260] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.490541 22672 raft_consensus.cc:2743] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:26.490180 22145 raft_consensus.cc:357] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.490973 22145 raft_consensus.cc:383] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.491207 22145 raft_consensus.cc:738] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.487139 22376 tablet_replica.cc:331] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.492172 22376 raft_consensus.cc:2238] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.492774 22376 raft_consensus.cc:2267] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:26.493749 21690 leader_election.cc:343] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.494007 22144 tablet_bootstrap.cc:492] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:26.494455 22144 ts_tablet_manager.cc:1397] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.260s user 0.009s sys 0.002s
I20250112 02:57:26.494537 22376 ts_tablet_manager.cc:1905] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.495440 21690 leader_election.cc:304] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:26.495141 22143 ts_tablet_manager.cc:1428] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250112 02:57:26.497010 22144 raft_consensus.cc:357] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.497617 22144 raft_consensus.cc:383] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.497846 22144 raft_consensus.cc:738] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.498250 22672 raft_consensus.cc:2743] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:26.498662 22144 consensus_queue.cc:260] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.499889 21690 leader_election.cc:304] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
W20250112 02:57:26.501571 21690 raft_consensus.cc:2681] T 301c8a2c534a4c54bfd6e678f3184ea4 P 1c2770701ca54288956df9d910d7d3d2: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250112 02:57:26.503762 21690 leader_election.cc:304] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.500594 22143 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:26.457870 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 139) took 2617 ms. Trace:
I20250112 02:57:26.510751 22672 raft_consensus.cc:2798] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.511307 22672 raft_consensus.cc:491] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.511705 22672 raft_consensus.cc:3054] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.512679 21386 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_732417b2b7f1413782c92f4849089b46" } new_table_name: "default.Table_e5f44414856346b8a9b001fe2d81f67f" modify_external_catalogs: true
I20250112 02:57:26.515589 22379 rpcz_store.cc:276] 0112 02:57:23.840707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.840841 (+ 134us) service_pool.cc:225] Handling call
0112 02:57:26.457412 (+2616571us) ts_tablet_manager.cc:1145] Deleting tablet 84f7700b8cce4261a32f2009d1481b99
0112 02:57:26.457469 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.457793 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2616037}
W20250112 02:57:26.517863 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f7700b8cce4261a32f2009d1481b99 already in progress: creating tablet
I20250112 02:57:26.518813 21392 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_256609c2ce914c68ad16988dc5dd7ca2" } modify_external_catalogs: true
I20250112 02:57:26.520084 22672 raft_consensus.cc:513] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.492194 22145 consensus_queue.cc:260] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.524173 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:26.525609 21691 leader_election.cc:343] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.519345 21392 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_256609c2ce914c68ad16988dc5dd7ca2" } modify_external_catalogs: true
I20250112 02:57:26.527630 22672 leader_election.cc:290] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.535327 22379 tablet_replica.cc:331] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:26.536089 22379 raft_consensus.cc:2238] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.536500 22379 raft_consensus.cc:2267] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:26.468952 21384 rpcz_store.cc:269] 0112 02:57:25.282177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.282295 (+ 118us) service_pool.cc:225] Handling call
0112 02:57:26.468586 (+1186291us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:26.219854 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:26.220231 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:26.220704 (+ 473us) write_op.cc:435] Acquired schema lock
0112 02:57:26.220720 (+ 16us) tablet.cc:659] Decoding operations
0112 02:57:26.220984 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:26.221027 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:26.221050 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:26.221226 (+ 176us) tablet.cc:698] Row locks acquired
0112 02:57:26.221241 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:26.221349 (+ 108us) write_op.cc:270] Start()
0112 02:57:26.221433 (+ 84us) write_op.cc:276] Timestamp: P: 1736650646221395 usec, L: 0
0112 02:57:26.221449 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:26.221922 (+ 473us) log.cc:844] Serialized 536 byte log entry
0112 02:57:26.267267 (+ 45345us) op_driver.cc:464] REPLICATION: finished
0112 02:57:26.267660 (+ 393us) write_op.cc:301] APPLY: starting
0112 02:57:26.267713 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:26.267794 (+ 81us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:26.267906 (+ 112us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:26.268053 (+ 147us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:26.268176 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:26.268221 (+ 45us) write_op.cc:312] APPLY: finished
0112 02:57:26.268393 (+ 172us) log.cc:844] Serialized 23 byte log entry
0112 02:57:26.268517 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:26.268576 (+ 59us) write_op.cc:454] Released schema lock
0112 02:57:26.268605 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:26.268832 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:26.268857 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":247,"apply.run_cpu_time_us":1137,"apply.run_wall_time_us":1136,"num_ops":1,"prepare.queue_time_us":154,"prepare.run_cpu_time_us":2396,"prepare.run_wall_time_us":3575,"raft.queue_time_us":1052,"raft.run_cpu_time_us":590,"raft.run_wall_time_us":591,"replication_time_us":45722}]]}
I20250112 02:57:26.538828 22145 ts_tablet_manager.cc:1428] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20250112 02:57:26.539634 22145 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.545459 21902 raft_consensus.cc:3054] T 22aac4332d0e476d9758004972ef8ab2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:26.545548 21980 consensus_peers.cc:487] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bca1c68aae24a1f9974a0bd2244fbf8. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:26.547885 22144 ts_tablet_manager.cc:1428] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.053s user 0.004s sys 0.001s
I20250112 02:57:26.549476 22145 tablet_bootstrap.cc:654] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.553437 22644 consensus_queue.cc:1035] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:26.538473 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_c562ca46d2f349e58930eef499aef1cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:26.555968 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c562ca46d2f349e58930eef499aef1cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.557745 22581 raft_consensus.cc:491] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cca160b7cffb4ae287ea077e7352ffc9)
I20250112 02:57:26.558333 22581 raft_consensus.cc:513] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.560586 22581 leader_election.cc:290] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:26.562685 21980 leader_election.cc:343] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Not found: Tablet not found: 77d80c6a60bc46449c7c0656128100c8
W20250112 02:57:26.565768 21978 leader_election.cc:343] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:26.566268 21978 leader_election.cc:304] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.567116 22581 raft_consensus.cc:2743] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:26.572815 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:26.573382 22042 raft_consensus.cc:3054] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.574112 22379 ts_tablet_manager.cc:1905] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.576753 22143 tablet_bootstrap.cc:654] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:26.582743 21385 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a consensus state has no leader
I20250112 02:57:26.582567 22750 tablet.cc:1721] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:26.584957 22144 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:26.607353 22644 consensus_queue.cc:1035] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:26.608204 22042 raft_consensus.cc:2463] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:26.609876 21690 leader_election.cc:304] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.611261 22645 raft_consensus.cc:2798] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.611825 22645 raft_consensus.cc:695] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:26.612823 22645 consensus_queue.cc:237] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.627314 22375 ts_tablet_manager.cc:1918] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:26.627774 22375 log.cc:1198] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/3c4998c034e54b1bae0ef132cef5b04d
I20250112 02:57:26.628592 22375 ts_tablet_manager.cc:1939] T 3c4998c034e54b1bae0ef132cef5b04d P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:26.630617 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 165) took 1219 ms. Trace:
I20250112 02:57:26.631001 22375 rpcz_store.cc:276] 0112 02:57:25.411173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.411289 (+ 116us) service_pool.cc:225] Handling call
0112 02:57:26.186860 (+775571us) ts_tablet_manager.cc:1145] Deleting tablet 3c4998c034e54b1bae0ef132cef5b04d
0112 02:57:26.186915 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.451233 (+264318us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.581248 (+130015us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.630565 (+ 49317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":774952}
I20250112 02:57:26.641019 22749 tablet.cc:1721] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:26.644150 22145 tablet_bootstrap.cc:492] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.644505 22145 ts_tablet_manager.cc:1397] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.008s
I20250112 02:57:26.646612 22145 raft_consensus.cc:357] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.647297 22145 raft_consensus.cc:383] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.647538 22145 raft_consensus.cc:738] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.648176 22145 consensus_queue.cc:260] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.650736 22376 ts_tablet_manager.cc:1918] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:26.650818 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:26.651070 22376 log.cc:1198] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/301c8a2c534a4c54bfd6e678f3184ea4
I20250112 02:57:26.651865 22376 ts_tablet_manager.cc:1939] T 301c8a2c534a4c54bfd6e678f3184ea4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:26.651818 22457 raft_consensus.cc:491] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.652186 22145 ts_tablet_manager.cc:1428] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250112 02:57:26.652287 22457 raft_consensus.cc:513] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.654567 22457 leader_election.cc:290] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:26.654831 21386 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13edf2c56e8e4254b75e0eb85fb88427 has no consensus state
I20250112 02:57:26.656244 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 142) took 3384 ms. Trace:
I20250112 02:57:26.656535 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:26.657140 21757 raft_consensus.cc:2463] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:26.656698 22376 rpcz_store.cc:276] 0112 02:57:23.271805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.271932 (+ 127us) service_pool.cc:225] Handling call
0112 02:57:26.487002 (+3215070us) ts_tablet_manager.cc:1145] Deleting tablet 301c8a2c534a4c54bfd6e678f3184ea4
0112 02:57:26.487038 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.535974 (+ 48936us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.650534 (+114560us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.656169 (+ 5635us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3214561}
I20250112 02:57:26.658046 21978 leader_election.cc:304] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:26.658345 22145 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.658726 22457 raft_consensus.cc:2798] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.659065 22457 raft_consensus.cc:491] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.659387 22457 raft_consensus.cc:3054] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.658046 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 301c8a2c534a4c54bfd6e678f3184ea4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]) successfully deleted
I20250112 02:57:26.661021 21387 catalog_manager.cc:6596] ReplaceTablet: tablet 6fbe0e20a157479e9160fc6782e3bf37 deleted and replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5
I20250112 02:57:26.661082 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:26.662696 21728 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:26.668201 22143 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:26.668645 22143 ts_tablet_manager.cc:1397] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.002s
I20250112 02:57:26.669651 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]) successfully deleted
I20250112 02:57:26.637212 22144 tablet_bootstrap.cc:654] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.657087 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:26.670946 22376 tablet_replica.cc:331] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.671603 22376 raft_consensus.cc:2238] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.672027 22376 raft_consensus.cc:2267] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.673629 22376 ts_tablet_manager.cc:1905] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.678176 22143 raft_consensus.cc:357] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.679520 22143 raft_consensus.cc:383] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.679646 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:26.679841 22143 raft_consensus.cc:738] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.680755 22145 tablet_bootstrap.cc:654] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:26.681464 21980 leader_election.cc:343] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:26.683167 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 443aa544f6df4d50a78c9effd306e216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443aa544f6df4d50a78c9effd306e216 already in progress: creating tablet
I20250112 02:57:26.683409 22143 consensus_queue.cc:260] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.686605 22143 ts_tablet_manager.cc:1428] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20250112 02:57:26.666059 22457 raft_consensus.cc:513] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.687403 22143 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:26.689756 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 994e1d114e5a4de5aa1d7651907d718e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994e1d114e5a4de5aa1d7651907d718e already in progress: creating tablet
I20250112 02:57:26.697207 22756 tablet.cc:1721] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250112 02:57:26.702001 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e86223dc27604b0488bcaf442de062a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:26.703593 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e86223dc27604b0488bcaf442de062a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.707443 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_d498875b28e14a3d815a24d80cb3cc7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.717474 22145 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.717881 22145 ts_tablet_manager.cc:1397] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.005s
I20250112 02:57:26.720055 22145 raft_consensus.cc:357] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.720152 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ff23deca1f7543abbaab1dcbea953bab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:26.720674 22145 raft_consensus.cc:383] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.720899 22145 raft_consensus.cc:738] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.723894 22376 ts_tablet_manager.cc:1918] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:26.724311 22376 log.cc:1198] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:26.725167 22376 ts_tablet_manager.cc:1939] T 77d80c6a60bc46449c7c0656128100c8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:26.726327 22145 consensus_queue.cc:260] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:26.728148 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff23deca1f7543abbaab1dcbea953bab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:26.728643 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d498875b28e14a3d815a24d80cb3cc7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:26.730902 22457 leader_election.cc:290] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:26.731017 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:26.731549 21757 raft_consensus.cc:3054] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.733330 22145 ts_tablet_manager.cc:1428] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250112 02:57:26.740957 22645 raft_consensus.cc:491] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.738544 22145 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.763269 22457 raft_consensus.cc:491] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.765580 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 150) took 2953 ms. Trace:
I20250112 02:57:26.765915 22376 rpcz_store.cc:276] 0112 02:57:23.812303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.812433 (+ 130us) service_pool.cc:225] Handling call
0112 02:57:26.670816 (+2858383us) ts_tablet_manager.cc:1145] Deleting tablet 77d80c6a60bc46449c7c0656128100c8
0112 02:57:26.670852 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.697717 (+ 26865us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.723645 (+ 25928us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.765525 (+ 41880us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2857843}
I20250112 02:57:26.773902 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 77d80c6a60bc46449c7c0656128100c8 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]) successfully deleted
I20250112 02:57:26.764654 22457 raft_consensus.cc:513] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:26.776533 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 already in progress: creating tablet
I20250112 02:57:26.779152 22145 tablet_bootstrap.cc:654] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.785609 21383 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_703a5763f259457bad2f7cbc579160be" } new_table_name: "default.Table_5ac32f685843445fb0c4c65fc92dd2ca" modify_external_catalogs: true
I20250112 02:57:26.794864 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:26.796027 21978 leader_election.cc:343] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.798054 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 151) took 2958 ms. Trace:
I20250112 02:57:26.798480 22376 rpcz_store.cc:276] 0112 02:57:23.839888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:23.839995 (+ 107us) service_pool.cc:225] Handling call
0112 02:57:26.774308 (+2934313us) ts_tablet_manager.cc:1145] Deleting tablet 84f7700b8cce4261a32f2009d1481b99
0112 02:57:26.797624 (+ 23316us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.797975 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927645}
W20250112 02:57:26.800806 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84f7700b8cce4261a32f2009d1481b99 already in progress: creating tablet
I20250112 02:57:26.741364 22645 raft_consensus.cc:513] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.804733 22645 leader_election.cc:290] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.805649 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.806245 22042 raft_consensus.cc:2463] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:26.807170 22145 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:26.807530 22145 ts_tablet_manager.cc:1397] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.073s user 0.010s sys 0.000s
I20250112 02:57:26.809741 22145 raft_consensus.cc:357] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.810385 22145 raft_consensus.cc:383] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.810750 22145 raft_consensus.cc:738] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.811417 22145 consensus_queue.cc:260] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:26.798468 22457 leader_election.cc:290] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:26.717819 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:26.814755 22145 ts_tablet_manager.cc:1428] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20250112 02:57:26.815577 21980 leader_election.cc:343] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.747191 22585 raft_consensus.cc:491] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.816475 22585 raft_consensus.cc:513] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.805456 22143 tablet_bootstrap.cc:654] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:26.807405 21690 leader_election.cc:304] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:26.818224 22644 raft_consensus.cc:2798] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.818534 22644 raft_consensus.cc:491] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.818810 22644 raft_consensus.cc:3054] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.820009 22585 leader_election.cc:290] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:26.820202 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:26.822868 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.823460 22042 raft_consensus.cc:2463] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:26.824542 22379 ts_tablet_manager.cc:1918] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:26.824542 21836 leader_election.cc:304] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:26.824862 22379 log.cc:1198] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/99e81199f79d46d6922eb1c6805d25da
I20250112 02:57:26.825407 22379 ts_tablet_manager.cc:1939] T 99e81199f79d46d6922eb1c6805d25da P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:26.826122 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 149) took 2301 ms. Trace:
I20250112 02:57:26.828416 21392 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 600345a7b3e949199ce79570fd978e76
I20250112 02:57:26.829802 22644 raft_consensus.cc:513] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.840323 22376 tablet_replica.cc:331] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.840328 22524 raft_consensus.cc:2798] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.840687 22524 raft_consensus.cc:491] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.840991 22524 raft_consensus.cc:3054] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.841132 21753 raft_consensus.cc:2388] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:26.841133 22376 raft_consensus.cc:2238] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.841560 22376 raft_consensus.cc:2267] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.843262 22376 ts_tablet_manager.cc:1905] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.845017 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:26.847064 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:26.847569 22042 raft_consensus.cc:3054] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.855541 22042 raft_consensus.cc:2463] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:26.857050 21690 leader_election.cc:304] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:26.858300 22644 leader_election.cc:290] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.858563 22645 raft_consensus.cc:2798] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.862876 22645 raft_consensus.cc:695] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:26.863687 22645 consensus_queue.cc:237] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.832528 21757 raft_consensus.cc:2463] T c80b7813b50d4739a76776d5d16434f3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:26.870002 21978 leader_election.cc:304] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.871641 22730 raft_consensus.cc:2798] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.872052 22730 raft_consensus.cc:695] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:26.872778 22730 consensus_queue.cc:237] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.882277 22376 ts_tablet_manager.cc:1918] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:26.826383 22379 rpcz_store.cc:276] 0112 02:57:24.524978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.525096 (+ 118us) service_pool.cc:225] Handling call
0112 02:57:26.535140 (+2010044us) ts_tablet_manager.cc:1145] Deleting tablet 99e81199f79d46d6922eb1c6805d25da
0112 02:57:26.535230 (+ 90us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.683871 (+148641us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.824336 (+140465us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.826074 (+ 1738us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991941}
I20250112 02:57:26.887506 22376 log.cc:1198] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/99e81199f79d46d6922eb1c6805d25da
I20250112 02:57:26.887881 22644 raft_consensus.cc:491] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.888269 22376 ts_tablet_manager.cc:1939] T 99e81199f79d46d6922eb1c6805d25da P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:26.776134 22375 tablet_replica.cc:331] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:26.780689 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:26.888338 22644 raft_consensus.cc:513] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:26.889714 21980 leader_election.cc:343] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.890110 21980 leader_election.cc:304] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.788663 22144 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:26.805017 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:26.890816 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:26.887660 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 99e81199f79d46d6922eb1c6805d25da (table default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]) successfully deleted
I20250112 02:57:26.891834 22144 ts_tablet_manager.cc:1397] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.343s user 0.009s sys 0.001s
W20250112 02:57:26.892098 21691 leader_election.cc:343] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.894285 22144 raft_consensus.cc:357] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.894982 22144 raft_consensus.cc:383] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:26.895270 22144 raft_consensus.cc:738] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:26.895198 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.895800 22042 raft_consensus.cc:2463] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:26.895920 22144 consensus_queue.cc:260] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.897055 21690 leader_election.cc:304] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.898142 22457 raft_consensus.cc:2743] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:26.898095 22672 raft_consensus.cc:2798] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.898558 22672 raft_consensus.cc:491] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.898986 22672 raft_consensus.cc:3054] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.892125 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 150) took 2366 ms. Trace:
I20250112 02:57:26.907569 22379 rpcz_store.cc:276] 0112 02:57:24.525328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.525420 (+ 92us) service_pool.cc:225] Handling call
0112 02:57:26.890678 (+2365258us) ts_tablet_manager.cc:1145] Deleting tablet 12f556cd7d6f4e13a525d3b8d4207d94
0112 02:57:26.891903 (+ 1225us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.892079 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2364697}
I20250112 02:57:26.908406 22672 raft_consensus.cc:513] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.909253 22644 leader_election.cc:290] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.913340 22672 leader_election.cc:290] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:26.913555 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:26.914311 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
W20250112 02:57:26.914712 21691 leader_election.cc:343] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:26.915134 22042 raft_consensus.cc:3054] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:26.914290 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 12f556cd7d6f4e13a525d3b8d4207d94 because the tablet was not found. No further retry: Not found: Tablet not found: 12f556cd7d6f4e13a525d3b8d4207d94
I20250112 02:57:26.921025 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 151) took 2394 ms. Trace:
I20250112 02:57:26.921325 22379 rpcz_store.cc:276] 0112 02:57:24.526402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.526530 (+ 128us) service_pool.cc:225] Handling call
0112 02:57:26.915478 (+2388948us) ts_tablet_manager.cc:1145] Deleting tablet f9bb62ab3f274c29b2c5f9528890b816
0112 02:57:26.915809 (+ 331us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.920968 (+ 5159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2388358}
I20250112 02:57:26.923199 22144 ts_tablet_manager.cc:1428] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.031s user 0.004s sys 0.000s
I20250112 02:57:26.924450 22144 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:26.930792 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet f9bb62ab3f274c29b2c5f9528890b816 because the tablet was not found. No further retry: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816
I20250112 02:57:26.931569 22145 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.932018 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 152) took 2404 ms. Trace:
I20250112 02:57:26.932282 22379 rpcz_store.cc:276] 0112 02:57:24.527639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.527739 (+ 100us) service_pool.cc:225] Handling call
0112 02:57:26.931763 (+2404024us) ts_tablet_manager.cc:1145] Deleting tablet bb324cd3120e45449e0500c99338135d
0112 02:57:26.931810 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.931972 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2403448}
W20250112 02:57:26.933317 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet bb324cd3120e45449e0500c99338135d because the tablet was not found. No further retry: Not found: Tablet not found: bb324cd3120e45449e0500c99338135d
I20250112 02:57:26.936021 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 162) took 2332 ms. Trace:
I20250112 02:57:26.936354 22376 rpcz_store.cc:276] 0112 02:57:24.604174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.605357 (+ 1183us) service_pool.cc:225] Handling call
0112 02:57:26.800887 (+2195530us) ts_tablet_manager.cc:1145] Deleting tablet 99e81199f79d46d6922eb1c6805d25da
0112 02:57:26.840193 (+ 39306us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.859345 (+ 19152us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.882076 (+ 22731us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:26.935962 (+ 53886us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137366}
I20250112 02:57:26.937623 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 164) took 2316 ms. Trace:
I20250112 02:57:26.937912 22376 rpcz_store.cc:276] 0112 02:57:24.620981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.628000 (+ 7019us) service_pool.cc:225] Handling call
0112 02:57:26.937379 (+2309379us) ts_tablet_manager.cc:1145] Deleting tablet afeff74abcc643a5ad6565d9dcc40d62
0112 02:57:26.937418 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.937575 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267822}
I20250112 02:57:26.943107 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 165) took 2284 ms. Trace:
I20250112 02:57:26.943182 22375 raft_consensus.cc:2238] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:26.943499 22376 rpcz_store.cc:276] 0112 02:57:24.659087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.661608 (+ 2521us) service_pool.cc:225] Handling call
0112 02:57:26.942753 (+2281145us) ts_tablet_manager.cc:1145] Deleting tablet 5e3890251e644c1eb596de043b63491e
0112 02:57:26.942811 (+ 58us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.943056 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273022}
I20250112 02:57:26.944438 22376 tablet_replica.cc:331] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:26.944718 22375 raft_consensus.cc:2267] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.945144 22376 raft_consensus.cc:2238] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:26.945827 22376 pending_rounds.cc:62] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Trying to abort 1 pending ops.
I20250112 02:57:26.946013 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 99e81199f79d46d6922eb1c6805d25da (table default.Table_2cd613a4fd3a4a5da98e2bc94a7dd3ed [id=d4f4bf5b80c54288ac6e2e20e1df191b]) successfully deleted
I20250112 02:57:26.946050 22376 pending_rounds.cc:69] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321047433084928 op_type: NO_OP noop_request { }
I20250112 02:57:26.942420 22145 tablet_bootstrap.cc:654] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:26.946686 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because the tablet was not found. No further retry: Not found: Tablet not found: afeff74abcc643a5ad6565d9dcc40d62
W20250112 02:57:26.947206 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5e3890251e644c1eb596de043b63491e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e3890251e644c1eb596de043b63491e already in progress: creating tablet
I20250112 02:57:26.946390 22376 raft_consensus.cc:2883] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:26.949056 22376 raft_consensus.cc:2267] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.951217 22376 ts_tablet_manager.cc:1905] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:26.953470 21387 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_290ad6dcb11640c6b0ebeb8d46e6a3b0" } new_table_name: "default.Table_00c60840930f46c49bf80fa7e4489419" modify_external_catalogs: true
I20250112 02:57:26.954990 22762 raft_consensus.cc:491] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:26.955397 22762 raft_consensus.cc:513] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.957162 22762 leader_election.cc:290] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:26.959932 22042 raft_consensus.cc:2463] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:26.960428 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f0d6a5afd140a8af77a5a39dc39dd2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:26.960464 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f0d6a5afd140a8af77a5a39dc39dd2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:26.960988 21757 raft_consensus.cc:2463] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:26.961047 22041 raft_consensus.cc:2463] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:26.962086 21690 leader_election.cc:304] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:26.962993 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 153) took 2434 ms. Trace:
I20250112 02:57:26.963238 21836 leader_election.cc:304] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:26.963459 22672 raft_consensus.cc:2798] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:26.963907 22672 raft_consensus.cc:695] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:26.965381 22672 consensus_queue.cc:237] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.968137 21392 master_service.cc:946] ReplaceTablet: received request to replace tablet 36495fb158a440309eddedd3910a7ad3 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:26.968750 22762 raft_consensus.cc:2798] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:26.969131 22762 raft_consensus.cc:491] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:26.969513 22762 raft_consensus.cc:3054] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.963284 22379 rpcz_store.cc:276] 0112 02:57:24.528751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.628561 (+ 99810us) service_pool.cc:225] Handling call
0112 02:57:26.962704 (+2334143us) ts_tablet_manager.cc:1145] Deleting tablet 5890ebb3c55943d5a334d6c93fbdaf0e
0112 02:57:26.962764 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.962945 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2327298}
W20250112 02:57:26.973302 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5890ebb3c55943d5a334d6c93fbdaf0e because the tablet was not found. No further retry: Not found: Tablet not found: 5890ebb3c55943d5a334d6c93fbdaf0e
I20250112 02:57:26.974220 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 154) took 2338 ms. Trace:
W20250112 02:57:26.975011 21383 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1800d0eb7a9a48d295aca16607291add consensus state has no leader
I20250112 02:57:26.978487 22379 rpcz_store.cc:276] 0112 02:57:24.636058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.636234 (+ 176us) service_pool.cc:225] Handling call
0112 02:57:26.973955 (+2337721us) ts_tablet_manager.cc:1145] Deleting tablet 402c24e54ab040e29949b1e4efbe5ff3
0112 02:57:26.974007 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.974174 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2337120}
W20250112 02:57:26.984939 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 402c24e54ab040e29949b1e4efbe5ff3 because the tablet was not found. No further retry: Not found: Tablet not found: 402c24e54ab040e29949b1e4efbe5ff3
I20250112 02:57:26.985306 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 155) took 2323 ms. Trace:
I20250112 02:57:26.985617 22379 rpcz_store.cc:276] 0112 02:57:24.662273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.662367 (+ 94us) service_pool.cc:225] Handling call
0112 02:57:26.984996 (+2322629us) ts_tablet_manager.cc:1145] Deleting tablet 5e3890251e644c1eb596de043b63491e
0112 02:57:26.985042 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.985262 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322114}
W20250112 02:57:26.986542 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5e3890251e644c1eb596de043b63491e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e3890251e644c1eb596de043b63491e already in progress: creating tablet
I20250112 02:57:26.986263 22762 raft_consensus.cc:513] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:26.986856 22379 tablet_replica.cc:331] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:26.990327 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f0d6a5afd140a8af77a5a39dc39dd2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:26.990917 22043 raft_consensus.cc:3054] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:26.993675 22379 raft_consensus.cc:2238] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:26.994413 22379 raft_consensus.cc:2267] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:26.998024 22043 raft_consensus.cc:2463] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:26.999271 21836 leader_election.cc:304] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:27.000238 22585 raft_consensus.cc:2798] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:27.004106 22379 ts_tablet_manager.cc:1905] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.017769 22376 ts_tablet_manager.cc:1918] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:27.018215 22376 log.cc:1198] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/98aa732d27f144aaad088ecc072837e9
I20250112 02:57:27.018992 22376 ts_tablet_manager.cc:1939] T 98aa732d27f144aaad088ecc072837e9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:27.021713 22581 raft_consensus.cc:491] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.023708 22581 raft_consensus.cc:513] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.024873 22143 tablet_bootstrap.cc:492] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:27.025225 22143 ts_tablet_manager.cc:1397] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.338s user 0.013s sys 0.028s
W20250112 02:57:27.027230 21379 consensus_peers.cc:487] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd -> Peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Couldn't send request to peer e9ef739ef9dc460ca8287b1d2c6c2217. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250112 02:57:27.027647 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:27.028877 21978 leader_election.cc:343] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.026268 22581 leader_election.cc:290] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.030794 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.027576 22143 raft_consensus.cc:357] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.031842 22143 raft_consensus.cc:383] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.031867 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 166) took 2308 ms. Trace:
I20250112 02:57:27.033454 22143 raft_consensus.cc:738] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:27.033141 21980 leader_election.cc:343] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.034018 21980 leader_election.cc:304] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:27.034132 22143 consensus_queue.cc:260] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.035528 22581 raft_consensus.cc:2743] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.036741 22376 rpcz_store.cc:276] 0112 02:57:24.723902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.743039 (+ 19137us) service_pool.cc:225] Handling call
0112 02:57:26.944309 (+2201270us) ts_tablet_manager.cc:1145] Deleting tablet 98aa732d27f144aaad088ecc072837e9
0112 02:57:26.944344 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:26.996758 (+ 52414us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.017540 (+ 20782us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.031818 (+ 14278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2074319}
I20250112 02:57:27.038021 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 98aa732d27f144aaad088ecc072837e9 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:27.036408 22143 ts_tablet_manager.cc:1428] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250112 02:57:27.039731 22143 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:27.038334 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 167) took 1992 ms. Trace:
I20250112 02:57:27.040429 22376 rpcz_store.cc:276] 0112 02:57:25.046278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.046384 (+ 106us) service_pool.cc:225] Handling call
0112 02:57:27.038052 (+1991668us) ts_tablet_manager.cc:1145] Deleting tablet 96f98f6840e24f3c9d03937c1bbc00e7
0112 02:57:27.038086 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.038290 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1990957}
W20250112 02:57:27.049341 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f98f6840e24f3c9d03937c1bbc00e7 already in progress: creating tablet
I20250112 02:57:27.049785 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 168) took 1975 ms. Trace:
I20250112 02:57:27.050139 22376 rpcz_store.cc:276] 0112 02:57:25.074353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.078136 (+ 3783us) service_pool.cc:225] Handling call
0112 02:57:27.049454 (+1971318us) ts_tablet_manager.cc:1145] Deleting tablet f445a4bf43db4571b6d3a4afe8fe823a
0112 02:57:27.049497 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.049726 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970636}
I20250112 02:57:27.051234 22375 ts_tablet_manager.cc:1905] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.052204 22379 ts_tablet_manager.cc:1918] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:57:27.052911 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet f445a4bf43db4571b6d3a4afe8fe823a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f445a4bf43db4571b6d3a4afe8fe823a already in progress: creating tablet
I20250112 02:57:27.053356 22376 tablet_replica.cc:331] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:27.054080 22376 raft_consensus.cc:2238] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:27.054960 22376 raft_consensus.cc:2267] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.055364 22379 log.cc:1198] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/98aa732d27f144aaad088ecc072837e9
I20250112 02:57:27.056035 22379 ts_tablet_manager.cc:1939] T 98aa732d27f144aaad088ecc072837e9 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:27.058210 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 156) took 2381 ms. Trace:
I20250112 02:57:27.058522 22379 rpcz_store.cc:276] 0112 02:57:24.677193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.749007 (+ 71814us) service_pool.cc:225] Handling call
0112 02:57:26.986681 (+2237674us) ts_tablet_manager.cc:1145] Deleting tablet 98aa732d27f144aaad088ecc072837e9
0112 02:57:26.986735 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.019296 (+ 32561us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.052027 (+ 32731us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.058168 (+ 6141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2237138}
I20250112 02:57:27.059302 22376 ts_tablet_manager.cc:1905] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.068385 21391 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_33ea8d4190104eb8837824b516dd2314" } modify_external_catalogs: true
I20250112 02:57:27.068895 21391 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_33ea8d4190104eb8837824b516dd2314" } modify_external_catalogs: true
I20250112 02:57:27.069896 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 98aa732d27f144aaad088ecc072837e9 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:27.071504 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 157) took 2314 ms. Trace:
I20250112 02:57:27.071841 22379 rpcz_store.cc:276] 0112 02:57:24.757506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.757777 (+ 271us) service_pool.cc:225] Handling call
0112 02:57:27.070059 (+2312282us) ts_tablet_manager.cc:1145] Deleting tablet 96f98f6840e24f3c9d03937c1bbc00e7
0112 02:57:27.071214 (+ 1155us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.071457 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2311798}
W20250112 02:57:27.073618 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f98f6840e24f3c9d03937c1bbc00e7 already in progress: creating tablet
I20250112 02:57:27.073724 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 158) took 2175 ms. Trace:
I20250112 02:57:27.074127 22379 rpcz_store.cc:276] 0112 02:57:24.897857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:24.897957 (+ 100us) service_pool.cc:225] Handling call
0112 02:57:27.073363 (+2175406us) ts_tablet_manager.cc:1145] Deleting tablet f445a4bf43db4571b6d3a4afe8fe823a
0112 02:57:27.073419 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.073677 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2174356}
I20250112 02:57:27.075347 22379 tablet_replica.cc:331] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:27.076022 22379 raft_consensus.cc:2238] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.078526 22379 raft_consensus.cc:2267] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:27.079459 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet f445a4bf43db4571b6d3a4afe8fe823a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f445a4bf43db4571b6d3a4afe8fe823a already in progress: creating tablet
I20250112 02:57:27.082578 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_7992c5d054614324abbd4d5c2fc805c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:27.084054 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7992c5d054614324abbd4d5c2fc805c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.088306 22379 ts_tablet_manager.cc:1905] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.090183 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_844490fe80a24e1babe918449efb5c2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:27.091667 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_844490fe80a24e1babe918449efb5c2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.102033 22376 ts_tablet_manager.cc:1918] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:27.102345 22376 log.cc:1198] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/0bca1c68aae24a1f9974a0bd2244fbf8
I20250112 02:57:27.114207 22143 tablet_bootstrap.cc:654] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.057627 22585 raft_consensus.cc:695] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:27.121465 22376 ts_tablet_manager.cc:1939] T 0bca1c68aae24a1f9974a0bd2244fbf8 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:27.123279 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 169) took 1876 ms. Trace:
I20250112 02:57:27.118317 22585 consensus_queue.cc:237] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.128643 22376 rpcz_store.cc:276] 0112 02:57:25.247160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.247969 (+ 809us) service_pool.cc:225] Handling call
0112 02:57:27.053234 (+1805265us) ts_tablet_manager.cc:1145] Deleting tablet 0bca1c68aae24a1f9974a0bd2244fbf8
0112 02:57:27.053268 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.093342 (+ 40074us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.101845 (+ 8503us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.123226 (+ 21381us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1804375}
I20250112 02:57:27.129757 22376 tablet_replica.cc:331] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:27.130046 22145 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:27.129923 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5]) successfully deleted
I20250112 02:57:27.130383 22145 ts_tablet_manager.cc:1397] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.199s user 0.009s sys 0.015s
I20250112 02:57:27.130424 22376 raft_consensus.cc:2238] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.130909 22376 raft_consensus.cc:2267] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.132400 22376 ts_tablet_manager.cc:1905] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.132719 22145 raft_consensus.cc:357] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.133338 22145 raft_consensus.cc:383] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.133586 22145 raft_consensus.cc:738] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.134233 22145 consensus_queue.cc:260] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.136238 22145 ts_tablet_manager.cc:1428] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 02:57:27.136982 22145 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:27.157850 22762 leader_election.cc:290] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.158218 22145 tablet_bootstrap.cc:654] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:27.163758 21387 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a69123c4ab384e0380c62e52b9f3b42d consensus state has no leader
I20250112 02:57:27.164415 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 33f89398c7aa4f2a938a476db897c833 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:27.173430 22376 ts_tablet_manager.cc:1918] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:27.173796 22376 log.cc:1198] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:27.174368 22376 ts_tablet_manager.cc:1939] T 9543a0a0359e4b72bd30f1cb6290f767 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:27.176396 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 170) took 1927 ms. Trace:
I20250112 02:57:27.176651 22376 rpcz_store.cc:276] 0112 02:57:25.248604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.251114 (+ 2510us) service_pool.cc:225] Handling call
0112 02:57:27.129612 (+1878498us) ts_tablet_manager.cc:1145] Deleting tablet 9543a0a0359e4b72bd30f1cb6290f767
0112 02:57:27.129649 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.158810 (+ 29161us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.173235 (+ 14425us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.176345 (+ 3110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1839690}
I20250112 02:57:27.177527 22376 tablet_replica.cc:331] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:27.177692 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 9543a0a0359e4b72bd30f1cb6290f767 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:27.178258 22376 raft_consensus.cc:2238] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.178755 22376 raft_consensus.cc:2267] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.187430 22376 ts_tablet_manager.cc:1905] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.189033 22145 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:27.189410 22145 ts_tablet_manager.cc:1397] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20250112 02:57:27.153405 22379 ts_tablet_manager.cc:1918] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:27.189880 22379 log.cc:1198] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/0bca1c68aae24a1f9974a0bd2244fbf8
I20250112 02:57:27.190640 22379 ts_tablet_manager.cc:1939] T 0bca1c68aae24a1f9974a0bd2244fbf8 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:27.191740 22145 raft_consensus.cc:357] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.192348 22145 raft_consensus.cc:383] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.192435 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 159) took 1980 ms. Trace:
I20250112 02:57:27.154896 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f0d6a5afd140a8af77a5a39dc39dd2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:27.192646 22145 raft_consensus.cc:738] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.193019 21757 raft_consensus.cc:3054] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.193307 22145 consensus_queue.cc:260] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.195222 22145 ts_tablet_manager.cc:1428] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 02:57:27.195979 22145 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:26.930419 22144 tablet_bootstrap.cc:654] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.198571 22379 rpcz_store.cc:276] 0112 02:57:25.212158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.212259 (+ 101us) service_pool.cc:225] Handling call
0112 02:57:27.075214 (+1862955us) ts_tablet_manager.cc:1145] Deleting tablet 0bca1c68aae24a1f9974a0bd2244fbf8
0112 02:57:27.075259 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.116646 (+ 41387us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.153207 (+ 36561us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.192353 (+ 39146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862442}
I20250112 02:57:27.200470 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 0bca1c68aae24a1f9974a0bd2244fbf8 (table default.Table_7b171c9ac8164714bfeeb9226098e7ff [id=e6249c87d6424956aab5e1967874bce5]) successfully deleted
I20250112 02:57:27.200551 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 160) took 1987 ms. Trace:
I20250112 02:57:27.200827 22379 rpcz_store.cc:276] 0112 02:57:25.213072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.213154 (+ 82us) service_pool.cc:225] Handling call
0112 02:57:27.199761 (+1986607us) ts_tablet_manager.cc:1145] Deleting tablet 4166dcd1f9d74255b21810dd42b7358d
0112 02:57:27.200320 (+ 559us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.200501 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10624,"tablet-delete.queue_time_us":1986110}
W20250112 02:57:27.201809 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 4166dcd1f9d74255b21810dd42b7358d because the tablet was not found. No further retry: Not found: Tablet not found: 4166dcd1f9d74255b21810dd42b7358d
I20250112 02:57:27.204785 22379 tablet_replica.cc:331] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:27.205525 22379 raft_consensus.cc:2238] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:27.206215 22379 raft_consensus.cc:2267] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.207917 22379 ts_tablet_manager.cc:1905] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.204066 21757 raft_consensus.cc:2463] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:27.150017 22524 raft_consensus.cc:513] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.212738 21899 raft_consensus.cc:2388] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate cca160b7cffb4ae287ea077e7352ffc9 in this term.
I20250112 02:57:27.215018 22524 leader_election.cc:290] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.216365 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:27.223258 22145 tablet_bootstrap.cc:654] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.236874 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0977a10252840a381234f02bb7339b3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:27.237573 22043 raft_consensus.cc:2388] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:27.238642 21836 leader_election.cc:304] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:27.239346 22524 raft_consensus.cc:2743] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.256006 22145 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:27.256382 22145 ts_tablet_manager.cc:1397] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.000s
I20250112 02:57:27.259078 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 6e922b68d33b4a76a260816475ee6a3c (DEFAULT_TABLE table=default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.259078 22376 ts_tablet_manager.cc:1918] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:27.260187 22376 log.cc:1198] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/3c4998c034e54b1bae0ef132cef5b04d
I20250112 02:57:27.260213 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e922b68d33b4a76a260816475ee6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.260660 22457 raft_consensus.cc:491] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.260923 22376 ts_tablet_manager.cc:1939] T 3c4998c034e54b1bae0ef132cef5b04d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:27.261041 22457 raft_consensus.cc:513] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.262717 22457 leader_election.cc:290] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.263928 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.264451 21757 raft_consensus.cc:2463] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:27.264617 22143 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:27.264977 22143 ts_tablet_manager.cc:1397] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.226s user 0.004s sys 0.006s
I20250112 02:57:27.265384 21978 leader_election.cc:304] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:27.265959 22457 raft_consensus.cc:2798] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:27.266211 22457 raft_consensus.cc:491] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:27.266448 22457 raft_consensus.cc:3054] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.268571 21728 tablet_service.cc:1467] Processing CreateTablet for tablet 6e922b68d33b4a76a260816475ee6a3c (DEFAULT_TABLE table=default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.268904 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.269606 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e922b68d33b4a76a260816475ee6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.271674 22144 tablet_bootstrap.cc:492] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:27.272047 22144 ts_tablet_manager.cc:1397] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.348s user 0.005s sys 0.004s
I20250112 02:57:27.271735 22457 raft_consensus.cc:513] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.274572 22143 raft_consensus.cc:357] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.275409 22143 raft_consensus.cc:383] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.275648 22143 raft_consensus.cc:738] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.276455 22457 leader_election.cc:290] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.278039 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:27.278527 21757 raft_consensus.cc:3054] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.279948 22143 consensus_queue.cc:260] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:27.280460 21980 leader_election.cc:343] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.283751 22143 ts_tablet_manager.cc:1428] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20250112 02:57:27.284714 22375 ts_tablet_manager.cc:1918] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:27.285030 22375 log.cc:1198] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:27.285858 22143 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:27.286803 21757 raft_consensus.cc:2463] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:27.280345 22145 raft_consensus.cc:357] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.292255 21978 leader_election.cc:304] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:27.290966 22145 raft_consensus.cc:383] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.293196 22145 raft_consensus.cc:738] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.293867 22145 consensus_queue.cc:260] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.294142 21727 tablet_service.cc:1467] Processing CreateTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d (DEFAULT_TABLE table=default.Table_e0b968d5b3cb4e34abd163429926d88e [id=7b2020d7f2ae4cae91616b74c1410aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.295120 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.295907 22145 ts_tablet_manager.cc:1428] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250112 02:57:27.296628 22581 raft_consensus.cc:2798] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:27.296757 22145 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:27.309072 22379 ts_tablet_manager.cc:1918] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:27.311913 22379 log.cc:1198] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/afeff74abcc643a5ad6565d9dcc40d62
I20250112 02:57:27.314518 21735 tablet_service.cc:1467] Processing CreateTablet for tablet f3da552a973945e284a386304f182777 (DEFAULT_TABLE table=default.Table_2573b35f681a4ede8201db20c64d179e [id=80273bec17084713b06f8f7f511fe6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.316143 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3da552a973945e284a386304f182777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.322479 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 33f89398c7aa4f2a938a476db897c833
I20250112 02:57:27.323715 21387 catalog_manager.cc:6596] ReplaceTablet: tablet 33f89398c7aa4f2a938a476db897c833 deleted and replaced by tablet e861bab8dcef41328095954a46e00a96
I20250112 02:57:27.324224 22379 ts_tablet_manager.cc:1939] T afeff74abcc643a5ad6565d9dcc40d62 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:27.326529 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f89398c7aa4f2a938a476db897c833 with delete_type TABLET_DATA_DELETED (replaced by tablet e861bab8dcef41328095954a46e00a96) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:27.325234 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 171) took 1910 ms. Trace:
I20250112 02:57:27.330386 22376 rpcz_store.cc:276] 0112 02:57:25.415214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.437788 (+ 22574us) service_pool.cc:225] Handling call
0112 02:57:27.177395 (+1739607us) ts_tablet_manager.cc:1145] Deleting tablet 3c4998c034e54b1bae0ef132cef5b04d
0112 02:57:27.177433 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.227939 (+ 50506us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.258888 (+ 30949us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.325178 (+ 66290us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739039}
I20250112 02:57:27.332019 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_28bd98bfe0924b169986563d8f51478c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:27.334844 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28bd98bfe0924b169986563d8f51478c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.333706 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e0b968d5b3cb4e34abd163429926d88e" } new_table_name: "default.Table_60849e78a19643198da449366a8e6ed6" modify_external_catalogs: true
I20250112 02:57:27.340524 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 161) took 2122 ms. Trace:
I20250112 02:57:27.342501 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 685953572d45476ebaf2580fd5e4a3cd (DEFAULT_TABLE table=default.Table_78a32d16e1b242c094697398cc472077 [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.343535 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685953572d45476ebaf2580fd5e4a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.343801 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]). Expected version 3 got 0
I20250112 02:57:27.345288 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:27.349454 21731 tablet_service.cc:1467] Processing CreateTablet for tablet 2d793344526a404e8a737e7fb628e521 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.350369 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d793344526a404e8a737e7fb628e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.286559 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:27.290108 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 6e922b68d33b4a76a260816475ee6a3c (DEFAULT_TABLE table=default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.355854 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e922b68d33b4a76a260816475ee6a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.293365 21876 tablet_service.cc:1467] Processing CreateTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d (DEFAULT_TABLE table=default.Table_e0b968d5b3cb4e34abd163429926d88e [id=7b2020d7f2ae4cae91616b74c1410aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.359192 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.274237 22144 raft_consensus.cc:357] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.315153 21872 tablet_service.cc:1467] Processing CreateTablet for tablet f3da552a973945e284a386304f182777 (DEFAULT_TABLE table=default.Table_2573b35f681a4ede8201db20c64d179e [id=80273bec17084713b06f8f7f511fe6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.367194 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3da552a973945e284a386304f182777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.326937 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f89398c7aa4f2a938a476db897c833 with delete_type TABLET_DATA_DELETED (replaced by tablet e861bab8dcef41328095954a46e00a96) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:27.373409 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 2d793344526a404e8a737e7fb628e521 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.374255 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d793344526a404e8a737e7fb628e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.339198 22145 tablet_bootstrap.cc:654] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.375792 21734 tablet_service.cc:1467] Processing CreateTablet for tablet 78f03a1dfb604b33a0528c28ade287f2 (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.341346 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 685953572d45476ebaf2580fd5e4a3cd (DEFAULT_TABLE table=default.Table_78a32d16e1b242c094697398cc472077 [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.376856 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f03a1dfb604b33a0528c28ade287f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.377467 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685953572d45476ebaf2580fd5e4a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:27.378311 21690 consensus_peers.cc:487] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:27.379411 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 78f03a1dfb604b33a0528c28ade287f2 (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.380697 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f03a1dfb604b33a0528c28ade287f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.388223 22144 raft_consensus.cc:383] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.388628 22144 raft_consensus.cc:738] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.389341 22144 consensus_queue.cc:260] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.389694 22778 raft_consensus.cc:491] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.390120 22778 raft_consensus.cc:513] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.392618 22730 raft_consensus.cc:491] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.393026 22730 raft_consensus.cc:513] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:27.394905 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 96250.1us
I20250112 02:57:27.396591 22145 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:27.396967 22145 ts_tablet_manager.cc:1397] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.006s
I20250112 02:57:27.397313 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 13edf2c56e8e4254b75e0eb85fb88427 (DEFAULT_TABLE table=default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.398283 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13edf2c56e8e4254b75e0eb85fb88427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.399255 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 13edf2c56e8e4254b75e0eb85fb88427 (DEFAULT_TABLE table=default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.400206 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13edf2c56e8e4254b75e0eb85fb88427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.399144 22145 raft_consensus.cc:357] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.400406 22145 raft_consensus.cc:383] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.400640 22145 raft_consensus.cc:738] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.397585 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.404753 21875 tablet_service.cc:1467] Processing CreateTablet for tablet ae3ed0bdc2ec44998ee461d4b1254e91 (DEFAULT_TABLE table=default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.405678 21875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3ed0bdc2ec44998ee461d4b1254e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.409962 21726 tablet_service.cc:1467] Processing CreateTablet for tablet ae3ed0bdc2ec44998ee461d4b1254e91 (DEFAULT_TABLE table=default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.410959 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3ed0bdc2ec44998ee461d4b1254e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.410735 22145 consensus_queue.cc:260] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.352015 22457 raft_consensus.cc:491] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.413193 22457 raft_consensus.cc:513] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.417670 22144 ts_tablet_manager.cc:1428] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.122s user 0.005s sys 0.000s
I20250112 02:57:27.412869 22145 ts_tablet_manager.cc:1428] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
W20250112 02:57:27.423918 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 consensus state has no leader
I20250112 02:57:27.424383 22145 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:27.428717 21757 raft_consensus.cc:2463] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:27.429913 22145 tablet_bootstrap.cc:654] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.437979 22144 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:27.441265 22375 ts_tablet_manager.cc:1939] T 6fbe0e20a157479e9160fc6782e3bf37 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
W20250112 02:57:27.442327 21691 consensus_peers.cc:487] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:27.443770 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 5d017c05abe94657bfbd2b38329bba16 (DEFAULT_TABLE table=default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.444851 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d017c05abe94657bfbd2b38329bba16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.445648 22524 raft_consensus.cc:491] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.446062 22524 raft_consensus.cc:513] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.457551 22645 consensus_queue.cc:1035] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:27.462407 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 58c49391d2ed4c3fa84fa2d071b06094 (DEFAULT_TABLE table=default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.463639 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c49391d2ed4c3fa84fa2d071b06094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.465255 22145 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:27.465618 22145 ts_tablet_manager.cc:1397] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.000s
I20250112 02:57:27.466660 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 5d017c05abe94657bfbd2b38329bba16 (DEFAULT_TABLE table=default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.467552 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d017c05abe94657bfbd2b38329bba16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.467679 22145 raft_consensus.cc:357] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.468338 22145 raft_consensus.cc:383] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.468560 22145 raft_consensus.cc:738] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.469034 22145 consensus_queue.cc:260] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.479444 22524 leader_election.cc:290] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.341001 22379 rpcz_store.cc:276] 0112 02:57:25.218142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.218247 (+ 105us) service_pool.cc:225] Handling call
0112 02:57:27.204634 (+1986387us) ts_tablet_manager.cc:1145] Deleting tablet afeff74abcc643a5ad6565d9dcc40d62
0112 02:57:27.204680 (+ 46us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.234532 (+ 29852us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.308872 (+ 74340us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.340441 (+ 31569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1985931}
I20250112 02:57:27.483776 22379 tablet_replica.cc:331] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:27.483744 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet afeff74abcc643a5ad6565d9dcc40d62 (table default.Table_2ae6922ba3dd4bbabb88dd71998911a5 [id=e38ca9bde38b4cff80f7ae17a829bda9]) successfully deleted
I20250112 02:57:27.480022 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.495767 22784 raft_consensus.cc:491] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.496433 22784 raft_consensus.cc:513] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.497458 22375 tablet_replica.cc:331] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:27.498277 22375 raft_consensus.cc:2238] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.498466 22784 leader_election.cc:290] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.498908 22375 raft_consensus.cc:2267] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.502764 22524 raft_consensus.cc:491] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.503213 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.503201 22524 raft_consensus.cc:513] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.503621 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.505314 22524 leader_election.cc:290] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:27.505249 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.506017 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]) successfully deleted
I20250112 02:57:27.507122 22143 tablet_bootstrap.cc:654] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.510759 22379 raft_consensus.cc:2238] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:27.511440 22379 pending_rounds.cc:62] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Trying to abort 1 pending ops.
I20250112 02:57:27.511657 22379 pending_rounds.cc:69] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321047994667008 op_type: NO_OP noop_request { }
I20250112 02:57:27.511987 22379 raft_consensus.cc:2883] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:27.512321 22379 raft_consensus.cc:2267] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.514537 22379 ts_tablet_manager.cc:1905] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:27.519052 21835 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:27.500083 22730 leader_election.cc:290] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.495009 22778 leader_election.cc:290] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:27.532502 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 174) took 1742 ms. Trace:
I20250112 02:57:27.532866 22376 rpcz_store.cc:276] 0112 02:57:25.790335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.790473 (+ 138us) service_pool.cc:225] Handling call
0112 02:57:27.532156 (+1741683us) ts_tablet_manager.cc:1145] Deleting tablet 443aa544f6df4d50a78c9effd306e216
0112 02:57:27.532198 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.532439 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741171}
I20250112 02:57:27.533874 22376 tablet_replica.cc:331] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:27.534812 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c1affbaf247059ab65222e2e95bab" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:27.535568 22041 raft_consensus.cc:2388] T f08c1affbaf247059ab65222e2e95bab P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:27.540731 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.501214 22777 raft_consensus.cc:491] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.501760 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f89398c7aa4f2a938a476db897c833 with delete_type TABLET_DATA_DELETED (replaced by tablet e861bab8dcef41328095954a46e00a96) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:27.542048 22777 raft_consensus.cc:513] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.543308 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.544064 22777 leader_election.cc:290] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.544782 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.546731 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 (DEFAULT_TABLE table=default.Table_ff93d70a357e442dafed35cd75ae6152 [id=7747b35d16654e1188fa4474b6e55dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.548204 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247f0ef0ada44d9d957c8ff42b0da4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.555166 21391 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:27.558010 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 36495fb158a440309eddedd3910a7ad3
I20250112 02:57:27.559178 21392 catalog_manager.cc:6596] ReplaceTablet: tablet 36495fb158a440309eddedd3910a7ad3 deleted and replaced by tablet 9c31714d18414800845671d783ac8173
W20250112 02:57:27.566057 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be5f2a4f6a8e4a64a7642d5a18b58a6d consensus state has no leader
I20250112 02:57:27.550062 21727 tablet_service.cc:1467] Processing CreateTablet for tablet be0b4a02597842d3ae1ad0c6197cc808 (DEFAULT_TABLE table=default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.501893 22457 leader_election.cc:290] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.501758 22008 tablet_service.cc:1467] Processing CreateTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d (DEFAULT_TABLE table=default.Table_e0b968d5b3cb4e34abd163429926d88e [id=7b2020d7f2ae4cae91616b74c1410aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.508824 22007 tablet_service.cc:1467] Processing CreateTablet for tablet f3da552a973945e284a386304f182777 (DEFAULT_TABLE table=default.Table_2573b35f681a4ede8201db20c64d179e [id=80273bec17084713b06f8f7f511fe6dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.509012 22017 tablet_service.cc:1467] Processing CreateTablet for tablet 685953572d45476ebaf2580fd5e4a3cd (DEFAULT_TABLE table=default.Table_78a32d16e1b242c094697398cc472077 [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.510341 22010 tablet_service.cc:1467] Processing CreateTablet for tablet 2d793344526a404e8a737e7fb628e521 (DEFAULT_TABLE table=default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.512162 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 78f03a1dfb604b33a0528c28ade287f2 (DEFAULT_TABLE table=default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.513523 22006 tablet_service.cc:1467] Processing CreateTablet for tablet 13edf2c56e8e4254b75e0eb85fb88427 (DEFAULT_TABLE table=default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.515594 22023 tablet_service.cc:1467] Processing CreateTablet for tablet 5d017c05abe94657bfbd2b38329bba16 (DEFAULT_TABLE table=default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.515594 22019 tablet_service.cc:1467] Processing CreateTablet for tablet ae3ed0bdc2ec44998ee461d4b1254e91 (DEFAULT_TABLE table=default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.517231 22009 tablet_service.cc:1467] Processing CreateTablet for tablet 58c49391d2ed4c3fa84fa2d071b06094 (DEFAULT_TABLE table=default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:27.517879 21980 leader_election.cc:343] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.573958 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:27.574204 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 56386.7us
I20250112 02:57:27.574851 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be0b4a02597842d3ae1ad0c6197cc808. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:27.574784 21980 leader_election.cc:343] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.575069 22009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c49391d2ed4c3fa84fa2d071b06094. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:27.575570 21980 leader_election.cc:343] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:27.576071 21980 leader_election.cc:343] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.577371 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.578627 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]) successfully deleted
I20250112 02:57:27.578995 22010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d793344526a404e8a737e7fb628e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.580610 21978 leader_election.cc:304] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:27.581244 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 443aa544f6df4d50a78c9effd306e216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443aa544f6df4d50a78c9effd306e216 already in progress: creating tablet
W20250112 02:57:27.581490 21978 leader_election.cc:343] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.581804 21978 leader_election.cc:304] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:27.582937 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78f03a1dfb604b33a0528c28ade287f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.584357 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.519634 22581 raft_consensus.cc:695] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:27.585711 22581 consensus_queue.cc:237] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.586928 22006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13edf2c56e8e4254b75e0eb85fb88427. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.587551 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae3ed0bdc2ec44998ee461d4b1254e91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.591532 22017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685953572d45476ebaf2580fd5e4a3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.594753 22023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d017c05abe94657bfbd2b38329bba16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.598783 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3da552a973945e284a386304f182777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.607303 22782 raft_consensus.cc:2798] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:27.607653 22782 raft_consensus.cc:491] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:27.607937 22782 raft_consensus.cc:3054] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.609828 22379 ts_tablet_manager.cc:1918] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:27.610158 22379 log.cc:1198] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:27.611236 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:27.611142 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 (DEFAULT_TABLE table=default.Table_ff93d70a357e442dafed35cd75ae6152 [id=7747b35d16654e1188fa4474b6e55dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.611474 22015 tablet_service.cc:1467] Processing CreateTablet for tablet be0b4a02597842d3ae1ad0c6197cc808 (DEFAULT_TABLE table=default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.612164 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247f0ef0ada44d9d957c8ff42b0da4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.612265 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:27.613158 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be0b4a02597842d3ae1ad0c6197cc808. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:27.613669 21980 leader_election.cc:343] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:27.614683 21980 leader_election.cc:343] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.615229 22787 raft_consensus.cc:2743] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.616814 22524 raft_consensus.cc:491] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.617229 22524 raft_consensus.cc:513] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.619148 22524 leader_election.cc:290] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.631655 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.632328 21757 raft_consensus.cc:2463] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:27.633684 21835 leader_election.cc:304] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:27.642416 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:27.642575 21873 tablet_service.cc:1467] Processing CreateTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 (DEFAULT_TABLE table=default.Table_ff93d70a357e442dafed35cd75ae6152 [id=7747b35d16654e1188fa4474b6e55dce]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:27.623173 21978 leader_election.cc:343] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.644335 21978 leader_election.cc:304] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:27.645323 21978 leader_election.cc:343] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.645726 21978 leader_election.cc:304] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:27.649729 22379 ts_tablet_manager.cc:1939] T 9543a0a0359e4b72bd30f1cb6290f767 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:27.650521 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 162) took 2147 ms. Trace:
I20250112 02:57:27.650797 22379 rpcz_store.cc:276] 0112 02:57:25.571012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.571104 (+ 92us) service_pool.cc:225] Handling call
0112 02:57:27.483601 (+1912497us) ts_tablet_manager.cc:1145] Deleting tablet 9543a0a0359e4b72bd30f1cb6290f767
0112 02:57:27.483663 (+ 62us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.569908 (+ 86245us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.609627 (+ 39719us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.650446 (+ 40819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1908594}
I20250112 02:57:27.651751 22379 tablet_replica.cc:331] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:27.649577 21385 catalog_manager.cc:5526] T 22aac4332d0e476d9758004972ef8ab2 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:27.649981 22778 raft_consensus.cc:2743] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.667894 22379 raft_consensus.cc:2238] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.643360 21881 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:27.668299 22379 pending_rounds.cc:62] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Trying to abort 1 pending ops.
I20250112 02:57:27.668510 22379 pending_rounds.cc:69] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7113321045333995520 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "3c4998c034e54b1bae0ef132cef5b04d" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c" dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" new_extra_config { } }
I20250112 02:57:27.669800 22379 raft_consensus.cc:2267] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.672195 21873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 247f0ef0ada44d9d957c8ff42b0da4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.672760 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 58c49391d2ed4c3fa84fa2d071b06094 (DEFAULT_TABLE table=default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.673098 22672 consensus_queue.cc:1035] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:27.674036 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58c49391d2ed4c3fa84fa2d071b06094. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.549054 22376 raft_consensus.cc:2238] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.682793 21880 tablet_service.cc:1467] Processing CreateTablet for tablet be0b4a02597842d3ae1ad0c6197cc808 (DEFAULT_TABLE table=default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.683607 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be0b4a02597842d3ae1ad0c6197cc808. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.686776 22457 raft_consensus.cc:2743] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.688009 22144 tablet_bootstrap.cc:654] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.687616 22379 ts_tablet_manager.cc:1905] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.648502 22375 ts_tablet_manager.cc:1905] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.651252 22770 raft_consensus.cc:2798] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:27.664070 21836 leader_election.cc:304] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:27.697639 22770 raft_consensus.cc:491] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:27.697980 22770 raft_consensus.cc:3054] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:27.698292 21836 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:27.698734 21836 leader_election.cc:304] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:27.702178 22769 raft_consensus.cc:3054] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.704222 22770 raft_consensus.cc:513] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.707355 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:27.707864 21757 raft_consensus.cc:3054] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.708498 21875 tablet_service.cc:1467] Processing CreateTablet for tablet 030d48c4421e45f0900705c4b6a4f4ed (DEFAULT_TABLE table=default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.709466 21875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030d48c4421e45f0900705c4b6a4f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.710692 22770 leader_election.cc:290] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.711220 22770 raft_consensus.cc:2743] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.715248 22376 raft_consensus.cc:2267] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.716040 21757 raft_consensus.cc:2463] T 1f21e3c769364b58bd371aac338db1f2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:27.717635 22015 tablet_service.cc:1467] Processing CreateTablet for tablet f780cf38ac4d41d8b59a3d8882a10030 (DEFAULT_TABLE table=default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.718533 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:27.719080 22044 raft_consensus.cc:2463] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:27.722041 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f780cf38ac4d41d8b59a3d8882a10030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.717034 22376 ts_tablet_manager.cc:1905] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.723564 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 030d48c4421e45f0900705c4b6a4f4ed (DEFAULT_TABLE table=default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.724447 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030d48c4421e45f0900705c4b6a4f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.735958 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f21e3c769364b58bd371aac338db1f2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:27.736485 22041 raft_consensus.cc:3054] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.699342 22777 raft_consensus.cc:491] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.741359 22777 raft_consensus.cc:513] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.744114 22041 raft_consensus.cc:2463] T 1f21e3c769364b58bd371aac338db1f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:27.713972 22769 raft_consensus.cc:2743] T f08c1affbaf247059ab65222e2e95bab P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:27.716991 21835 leader_election.cc:304] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:27.745414 22769 raft_consensus.cc:2798] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:27.739154 21879 tablet_service.cc:1467] Processing CreateTablet for tablet f780cf38ac4d41d8b59a3d8882a10030 (DEFAULT_TABLE table=default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:27.748569 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f780cf38ac4d41d8b59a3d8882a10030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:27.755759 22769 raft_consensus.cc:695] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:27.756733 22769 consensus_queue.cc:237] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:27.781105 21980 consensus_peers.cc:487] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:27.784806 22379 ts_tablet_manager.cc:1918] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:27.788010 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_1b38015db69648f3a3fb3dc1c096a9e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:27.788232 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_0d55eaca1ca1484a8b8840ad492e7529"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:27.789455 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b38015db69648f3a3fb3dc1c096a9e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.791136 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_9e4986239f004d628ff28ec47d296db4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:27.792428 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e4986239f004d628ff28ec47d296db4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.793605 22376 ts_tablet_manager.cc:1918] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:27.794646 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_844490fe80a24e1babe918449efb5c2f" } new_table_name: "default.Table_7d88c7e3065640629e9d17b64174c9e4" modify_external_catalogs: true
I20250112 02:57:27.795420 22379 log.cc:1198] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/3c4998c034e54b1bae0ef132cef5b04d
I20250112 02:57:27.796061 22379 ts_tablet_manager.cc:1939] T 3c4998c034e54b1bae0ef132cef5b04d P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:27.796759 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 163) took 2225 ms. Trace:
I20250112 02:57:27.797011 22379 rpcz_store.cc:276] 0112 02:57:25.571333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.571431 (+ 98us) service_pool.cc:225] Handling call
0112 02:57:27.651609 (+2080178us) ts_tablet_manager.cc:1145] Deleting tablet 3c4998c034e54b1bae0ef132cef5b04d
0112 02:57:27.651652 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.727246 (+ 75594us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.784610 (+ 57364us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.796714 (+ 12104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83,"tablet-delete.queue_time_us":2076326}
I20250112 02:57:27.798133 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 166) took 1781 ms. Trace:
I20250112 02:57:27.798395 22379 rpcz_store.cc:276] 0112 02:57:26.017171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.022707 (+ 5536us) service_pool.cc:225] Handling call
0112 02:57:27.797863 (+1775156us) ts_tablet_manager.cc:1145] Deleting tablet 443aa544f6df4d50a78c9effd306e216
0112 02:57:27.797894 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.798092 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774187}
I20250112 02:57:27.799399 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 167) took 1781 ms. Trace:
I20250112 02:57:27.799649 22379 rpcz_store.cc:276] 0112 02:57:26.017529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.023956 (+ 6427us) service_pool.cc:225] Handling call
0112 02:57:27.799091 (+1775135us) ts_tablet_manager.cc:1145] Deleting tablet 994e1d114e5a4de5aa1d7651907d718e
0112 02:57:27.799140 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.799358 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774252}
I20250112 02:57:27.800515 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 169) took 1467 ms. Trace:
I20250112 02:57:27.800755 22379 rpcz_store.cc:276] 0112 02:57:26.332987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.338589 (+ 5602us) service_pool.cc:225] Handling call
0112 02:57:27.800331 (+1461742us) ts_tablet_manager.cc:1145] Deleting tablet 5f787ffe65e34e8f9852d149fff9f947
0112 02:57:27.800362 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.800477 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461209}
I20250112 02:57:27.801760 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 168) took 1469 ms. Trace:
I20250112 02:57:27.802067 22379 rpcz_store.cc:276] 0112 02:57:26.332705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.384438 (+ 51733us) service_pool.cc:225] Handling call
0112 02:57:27.801494 (+1417056us) ts_tablet_manager.cc:1145] Deleting tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9
0112 02:57:27.801524 (+ 30us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.801721 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1416478}
I20250112 02:57:27.803267 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 171) took 1369 ms. Trace:
I20250112 02:57:27.803571 22379 rpcz_store.cc:276] 0112 02:57:26.433447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.433587 (+ 140us) service_pool.cc:225] Handling call
0112 02:57:27.803045 (+1369458us) ts_tablet_manager.cc:1145] Deleting tablet 301c8a2c534a4c54bfd6e678f3184ea4
0112 02:57:27.803076 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.803226 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368901}
I20250112 02:57:27.804543 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 170) took 1371 ms. Trace:
I20250112 02:57:27.804852 22379 rpcz_store.cc:276] 0112 02:57:26.432791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.432944 (+ 153us) service_pool.cc:225] Handling call
0112 02:57:27.804333 (+1371389us) ts_tablet_manager.cc:1145] Deleting tablet 33a5c293bbd643aab9409c6e9abe4f0a
0112 02:57:27.804366 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.804502 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369550}
I20250112 02:57:27.805826 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 173) took 1337 ms. Trace:
I20250112 02:57:27.806166 22379 rpcz_store.cc:276] 0112 02:57:26.468153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.468603 (+ 450us) service_pool.cc:225] Handling call
0112 02:57:27.805608 (+1337005us) ts_tablet_manager.cc:1145] Deleting tablet 7847b7af9e0b4aaba1f7cd06534f6510
0112 02:57:27.805642 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.805782 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336467}
I20250112 02:57:27.807358 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 174) took 1338 ms. Trace:
I20250112 02:57:27.807808 22379 rpcz_store.cc:276] 0112 02:57:26.468466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.469469 (+ 1003us) service_pool.cc:225] Handling call
0112 02:57:27.806998 (+1337529us) ts_tablet_manager.cc:1145] Deleting tablet 77d80c6a60bc46449c7c0656128100c8
0112 02:57:27.807049 (+ 51us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.807298 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1336966}
I20250112 02:57:27.808267 22376 log.cc:1198] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:27.808923 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 172) took 1341 ms. Trace:
I20250112 02:57:27.809250 22379 rpcz_store.cc:276] 0112 02:57:26.467591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.467724 (+ 133us) service_pool.cc:225] Handling call
0112 02:57:27.808706 (+1340982us) ts_tablet_manager.cc:1145] Deleting tablet 7d699356dd3341338670b433a7fa02ad
0112 02:57:27.808742 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.808884 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337652}
I20250112 02:57:27.810146 22379 tablet_replica.cc:331] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:27.810786 22379 raft_consensus.cc:2238] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:27.810997 22777 leader_election.cc:290] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:27.811291 22379 raft_consensus.cc:2267] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.812372 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:27.813129 22379 ts_tablet_manager.cc:1905] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:27.813609 21980 leader_election.cc:343] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.815342 22376 ts_tablet_manager.cc:1939] T 994e1d114e5a4de5aa1d7651907d718e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:27.825167 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 175) took 2001 ms. Trace:
I20250112 02:57:27.830591 22376 rpcz_store.cc:276] 0112 02:57:25.823336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.823462 (+ 126us) service_pool.cc:225] Handling call
0112 02:57:27.533745 (+1710283us) ts_tablet_manager.cc:1145] Deleting tablet 994e1d114e5a4de5aa1d7651907d718e
0112 02:57:27.533780 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.744567 (+210787us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.793410 (+ 48843us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:27.824806 (+ 31396us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47276,"tablet-delete.queue_time_us":1709767}
W20250112 02:57:27.836189 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d55eaca1ca1484a8b8840ad492e7529 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:27.837374 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 994e1d114e5a4de5aa1d7651907d718e (table default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]) successfully deleted
I20250112 02:57:27.841465 22782 raft_consensus.cc:513] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.844584 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:27.845728 21980 leader_election.cc:343] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.851068 22644 raft_consensus.cc:491] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.851491 22644 raft_consensus.cc:513] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.853590 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 9543a0a0359e4b72bd30f1cb6290f767 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:27.856164 22644 leader_election.cc:290] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:27.859272 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:27.863584 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 922b22f0b1284bc4a732371a61cb1ed4 has no consensus state
I20250112 02:57:27.878278 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:27.878856 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:27.879539 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:27.880112 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:27.846184 22782 leader_election.cc:290] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:27.894786 22144 tablet_bootstrap.cc:492] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:27.895253 22144 ts_tablet_manager.cc:1397] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.473s user 0.008s sys 0.002s
I20250112 02:57:27.898281 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 176) took 2032 ms. Trace:
I20250112 02:57:27.898636 22376 rpcz_store.cc:276] 0112 02:57:25.866152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.866273 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:27.834874 (+1968601us) ts_tablet_manager.cc:1145] Deleting tablet f9bb62ab3f274c29b2c5f9528890b816
0112 02:57:27.898058 (+ 63184us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.898229 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964432}
I20250112 02:57:27.899734 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 177) took 1815 ms. Trace:
I20250112 02:57:27.899998 22376 rpcz_store.cc:276] 0112 02:57:26.084761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.084923 (+ 162us) service_pool.cc:225] Handling call
0112 02:57:27.899431 (+1814508us) ts_tablet_manager.cc:1145] Deleting tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9
0112 02:57:27.899468 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.899687 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813952}
I20250112 02:57:27.900866 22376 tablet_replica.cc:331] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:27.901557 22376 raft_consensus.cc:2238] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250112 02:57:27.903144 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet f9bb62ab3f274c29b2c5f9528890b816 because the tablet was not found. No further retry: Not found: Tablet not found: f9bb62ab3f274c29b2c5f9528890b816
W20250112 02:57:27.903767 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 already in progress: creating tablet
I20250112 02:57:27.904940 22376 raft_consensus.cc:2267] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:27.905665 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_78a32d16e1b242c094697398cc472077" } new_table_name: "default.Table_9456a239683340bcb3674e78ab5db82b" modify_external_catalogs: true
I20250112 02:57:27.907752 22145 ts_tablet_manager.cc:1428] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.442s user 0.025s sys 0.024s
I20250112 02:57:27.908608 21384 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_8e51a35f779844a3a2e5fec4500f7697" } modify_external_catalogs: true
I20250112 02:57:27.909205 21384 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_8e51a35f779844a3a2e5fec4500f7697" } modify_external_catalogs: true
I20250112 02:57:27.913334 22376 ts_tablet_manager.cc:1905] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:27.923228 22145 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:27.917902 22143 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
W20250112 02:57:27.928803 21980 consensus_peers.cc:487] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:27.930922 22645 consensus_queue.cc:1035] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:27.931110 22143 ts_tablet_manager.cc:1397] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.645s user 0.013s sys 0.000s
I20250112 02:57:27.949249 22145 tablet_bootstrap.cc:654] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:27.936101 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:27.954578 22730 raft_consensus.cc:491] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:27.954993 22730 raft_consensus.cc:513] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.956826 22730 leader_election.cc:290] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:27.957882 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:27.960610 21980 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:27.961171 22144 raft_consensus.cc:357] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.961855 22144 raft_consensus.cc:383] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.966621 22041 raft_consensus.cc:2463] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:27.967937 21690 leader_election.cc:304] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:27.968773 22645 raft_consensus.cc:2798] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:27.969146 22645 raft_consensus.cc:491] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:27.969522 22645 raft_consensus.cc:3054] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.972415 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:27.973798 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65eb8c99917247edbdbf5de42f3b505c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:27.974304 21751 raft_consensus.cc:3054] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:27.979326 22143 raft_consensus.cc:357] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.986653 22143 raft_consensus.cc:383] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:27.981837 21751 raft_consensus.cc:2463] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:27.987128 22143 raft_consensus.cc:738] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:27.988770 22143 consensus_queue.cc:260] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:27.996860 22143 ts_tablet_manager.cc:1428] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20250112 02:57:28.011113 22143 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:28.013185 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:28.034879 22645 raft_consensus.cc:513] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:27.962910 22769 consensus_queue.cc:1035] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:28.057480 22145 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.058137 22145 ts_tablet_manager.cc:1397] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.135s user 0.015s sys 0.006s
I20250112 02:57:28.061381 22145 raft_consensus.cc:357] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.054857 22144 raft_consensus.cc:738] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.067032 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 3c4998c034e54b1bae0ef132cef5b04d (table default.Table_4cf73afc8c034d84bb7ffb3aeda70d5c [id=c3d3f895021c4a5a9d3b4649c3690a28]) successfully deleted
I20250112 02:57:28.070891 21384 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d62c9a20cb8444feb126090656ea32fc
I20250112 02:57:28.071741 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_be71acfdd72940f6b6f99347736902fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:28.073338 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be71acfdd72940f6b6f99347736902fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:28.075160 22145 raft_consensus.cc:383] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.075485 22145 raft_consensus.cc:738] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.076210 22145 consensus_queue.cc:260] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.078639 22145 ts_tablet_manager.cc:1428] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20250112 02:57:28.080410 22144 consensus_queue.cc:260] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.082073 22145 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.083042 21387 master_service.cc:946] ReplaceTablet: received request to replace tablet 36495fb158a440309eddedd3910a7ad3 from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:28.083949 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 685953572d45476ebaf2580fd5e4a3cd consensus state has no leader
I20250112 02:57:28.093115 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_5ac32f685843445fb0c4c65fc92dd2ca" } new_table_name: "default.Table_4dcdc6c673294f60a1b60f8517bded49" modify_external_catalogs: true
I20250112 02:57:28.098152 22145 tablet_bootstrap.cc:654] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:28.111543 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 443aa544f6df4d50a78c9effd306e216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 443aa544f6df4d50a78c9effd306e216 already in progress: creating tablet
W20250112 02:57:28.112378 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 994e1d114e5a4de5aa1d7651907d718e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994e1d114e5a4de5aa1d7651907d718e already in progress: creating tablet
W20250112 02:57:28.113147 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5f787ffe65e34e8f9852d149fff9f947 because the tablet was not found. No further retry: Not found: Tablet not found: 5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:28.116096 22145 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.116478 22145 ts_tablet_manager.cc:1397] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20250112 02:57:28.118562 22145 raft_consensus.cc:357] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.119226 22145 raft_consensus.cc:383] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.119462 22145 raft_consensus.cc:738] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.119989 22145 consensus_queue.cc:260] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.044304 22143 tablet_bootstrap.cc:654] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.047521 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:28.056095 21726 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:28.067006 21731 tablet_service.cc:1467] Processing CreateTablet for tablet f780cf38ac4d41d8b59a3d8882a10030 (DEFAULT_TABLE table=default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:28.124373 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f780cf38ac4d41d8b59a3d8882a10030. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:28.068189 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 030d48c4421e45f0900705c4b6a4f4ed (DEFAULT_TABLE table=default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:28.072219 22379 ts_tablet_manager.cc:1918] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.130687 22379 log.cc:1198] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:28.131021 22144 ts_tablet_manager.cc:1428] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.172s user 0.005s sys 0.001s
I20250112 02:57:28.131158 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 030d48c4421e45f0900705c4b6a4f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:28.131392 22379 ts_tablet_manager.cc:1939] T 6fbe0e20a157479e9160fc6782e3bf37 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:28.132136 22379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 175) took 1469 ms. Trace:
I20250112 02:57:28.132419 22379 rpcz_store.cc:276] 0112 02:57:26.662388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.662533 (+ 145us) service_pool.cc:225] Handling call
0112 02:57:27.810026 (+1147493us) ts_tablet_manager.cc:1145] Deleting tablet 6fbe0e20a157479e9160fc6782e3bf37
0112 02:57:27.810057 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:27.984765 (+174708us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.071886 (+ 87121us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.132085 (+ 60199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1146929}
I20250112 02:57:28.133401 22379 tablet_replica.cc:331] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:28.134049 22379 raft_consensus.cc:2238] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:28.134112 22144 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:28.134470 22379 raft_consensus.cc:2267] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:28.136247 22379 ts_tablet_manager.cc:1905] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:28.145460 22145 ts_tablet_manager.cc:1428] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250112 02:57:28.146236 22145 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.153815 22645 leader_election.cc:290] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.154778 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:28.155326 22040 raft_consensus.cc:3054] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:28.167848 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 already in progress: creating tablet
W20250112 02:57:28.168675 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 301c8a2c534a4c54bfd6e678f3184ea4 because the tablet was not found. No further retry: Not found: Tablet not found: 301c8a2c534a4c54bfd6e678f3184ea4
W20250112 02:57:28.169508 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 33a5c293bbd643aab9409c6e9abe4f0a because the tablet was not found. No further retry: Not found: Tablet not found: 33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:28.169961 22143 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:28.170394 22143 ts_tablet_manager.cc:1397] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.161s user 0.013s sys 0.005s
W20250112 02:57:28.170567 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 7847b7af9e0b4aaba1f7cd06534f6510 because the tablet was not found. No further retry: Not found: Tablet not found: 7847b7af9e0b4aaba1f7cd06534f6510
I20250112 02:57:28.135283 21386 catalog_manager.cc:5526] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:28.173736 22143 raft_consensus.cc:357] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.174360 22143 raft_consensus.cc:383] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.174643 22143 raft_consensus.cc:738] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.175462 22143 consensus_queue.cc:260] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:28.181326 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 77d80c6a60bc46449c7c0656128100c8 because the tablet was not found. No further retry: Not found: Tablet not found: 77d80c6a60bc46449c7c0656128100c8
I20250112 02:57:28.182019 22143 ts_tablet_manager.cc:1428] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250112 02:57:28.182260 22379 ts_tablet_manager.cc:1918] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250112 02:57:28.182423 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 7d699356dd3341338670b433a7fa02ad because the tablet was not found. No further retry: Not found: Tablet not found: 7d699356dd3341338670b433a7fa02ad
I20250112 02:57:28.182840 22143 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:28.183303 22145 tablet_bootstrap.cc:654] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.189283 21384 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_426ae97685174521adbbaa140dad7cdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:28.190964 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.194628 22379 log.cc:1198] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/33f89398c7aa4f2a938a476db897c833
I20250112 02:57:28.195328 22379 ts_tablet_manager.cc:1939] T 33f89398c7aa4f2a938a476db897c833 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:28.196768 22379 tablet_replica.cc:331] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:28.197417 22379 raft_consensus.cc:2238] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:28.197836 22379 raft_consensus.cc:2267] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:28.201201 22376 ts_tablet_manager.cc:1918] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.201535 22376 log.cc:1198] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:28.202204 22376 ts_tablet_manager.cc:1939] T 6fbe0e20a157479e9160fc6782e3bf37 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:28.204010 22145 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:28.204093 21978 leader_election.cc:343] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.204234 22581 raft_consensus.cc:491] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.204391 22145 ts_tablet_manager.cc:1397] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.000s
I20250112 02:57:28.204489 21978 leader_election.cc:304] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.204613 22581 raft_consensus.cc:513] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.205399 22782 raft_consensus.cc:2743] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.206573 22145 raft_consensus.cc:357] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.207175 22145 raft_consensus.cc:383] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.207923 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.207441 22581 leader_election.cc:290] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:28.209051 22379 ts_tablet_manager.cc:1905] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:28.207386 22145 raft_consensus.cc:738] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:28.209980 21980 leader_election.cc:343] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.210736 22145 consensus_queue.cc:260] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:28.219409 21978 leader_election.cc:343] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.219760 21978 leader_election.cc:304] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.220425 22581 raft_consensus.cc:2743] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.221854 21978 leader_election.cc:304] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.222111 22040 raft_consensus.cc:2463] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:28.222601 22581 raft_consensus.cc:2798] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:28.222944 22581 raft_consensus.cc:695] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:28.223568 22581 consensus_queue.cc:237] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.224542 22145 ts_tablet_manager.cc:1428] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250112 02:57:28.225354 22145 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.229435 21690 leader_election.cc:304] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.230469 22672 raft_consensus.cc:2798] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:28.231657 22672 raft_consensus.cc:695] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:28.232492 22672 consensus_queue.cc:237] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:28.236821 22375 ts_tablet_manager.cc:1918] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.237205 22375 log.cc:1198] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/33f89398c7aa4f2a938a476db897c833
I20250112 02:57:28.237860 22375 ts_tablet_manager.cc:1939] T 33f89398c7aa4f2a938a476db897c833 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:28.199227 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:28.249621 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_5ac32f685843445fb0c4c65fc92dd2ca [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 1 got 0
I20250112 02:57:28.250385 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:28.252420 22379 ts_tablet_manager.cc:1918] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.252768 22379 log.cc:1198] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/36495fb158a440309eddedd3910a7ad3
I20250112 02:57:28.253443 22379 ts_tablet_manager.cc:1939] T 36495fb158a440309eddedd3910a7ad3 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:28.226318 22376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53076 (request call id 180) took 1565 ms. Trace:
I20250112 02:57:28.254623 22145 tablet_bootstrap.cc:654] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.254634 22376 rpcz_store.cc:276] 0112 02:57:26.660838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.660959 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:27.900738 (+1239779us) ts_tablet_manager.cc:1145] Deleting tablet 6fbe0e20a157479e9160fc6782e3bf37
0112 02:57:27.900770 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:28.010726 (+109956us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.200990 (+190264us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.226268 (+ 25278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3628,"tablet-delete.queue_time_us":1239301}
I20250112 02:57:28.263814 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]) successfully deleted
I20250112 02:57:28.264132 22376 tablet_replica.cc:331] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:28.264812 22376 raft_consensus.cc:2238] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:28.265580 22376 raft_consensus.cc:2267] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:28.267127 22672 raft_consensus.cc:491] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.267590 22672 raft_consensus.cc:513] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:28.270993 21690 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:28.271448 22672 leader_election.cc:290] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.275321 22376 ts_tablet_manager.cc:1905] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:28.278489 22145 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.278885 22145 ts_tablet_manager.cc:1397] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20250112 02:57:28.281205 22145 raft_consensus.cc:357] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.281821 22145 raft_consensus.cc:383] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.282085 22145 raft_consensus.cc:738] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.282984 21383 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:28.283589 22770 consensus_queue.cc:1035] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.290705 22145 consensus_queue.cc:260] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.293082 22145 ts_tablet_manager.cc:1428] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250112 02:57:28.293838 22145 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.300622 22782 consensus_queue.cc:1035] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.318483 22143 tablet_bootstrap.cc:654] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.318482 22145 tablet_bootstrap.cc:654] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:28.326838 21978 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.327262 21978 leader_election.cc:304] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:28.329049 21691 consensus_peers.cc:487] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:28.329574 21691 leader_election.cc:343] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.330663 22581 raft_consensus.cc:2743] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:28.293805 21384 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_426ae97685174521adbbaa140dad7cdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:28.334414 22645 consensus_queue.cc:1035] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.335588 22645 consensus_queue.cc:1035] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.339756 22581 raft_consensus.cc:491] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.340225 22376 ts_tablet_manager.cc:1918] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.340613 22376 log.cc:1198] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/33f89398c7aa4f2a938a476db897c833
I20250112 02:57:28.341267 22376 ts_tablet_manager.cc:1939] T 33f89398c7aa4f2a938a476db897c833 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:28.340234 22581 raft_consensus.cc:513] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.344921 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.345778 22581 leader_election.cc:290] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:28.346107 21980 leader_election.cc:343] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.355976 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 33f89398c7aa4f2a938a476db897c833 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]) successfully deleted
I20250112 02:57:28.356457 22376 tablet_replica.cc:331] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:28.356667 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 2601daad8ba742ec88ccfe603440e8e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2601daad8ba742ec88ccfe603440e8e7 already in progress: creating tablet
I20250112 02:57:28.357218 22376 raft_consensus.cc:2238] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:28.357972 22376 raft_consensus.cc:2267] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:28.360206 22144 tablet_bootstrap.cc:654] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.360337 22145 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.360751 22145 ts_tablet_manager.cc:1397] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.000s
I20250112 02:57:28.362819 22145 raft_consensus.cc:357] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.363484 22145 raft_consensus.cc:383] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.363730 22145 raft_consensus.cc:738] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.366216 22143 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:28.366686 22143 ts_tablet_manager.cc:1397] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.184s user 0.013s sys 0.000s
I20250112 02:57:28.368707 22143 raft_consensus.cc:357] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.369493 22143 raft_consensus.cc:383] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.370764 22145 consensus_queue.cc:260] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.372592 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 6fbe0e20a157479e9160fc6782e3bf37 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]) successfully deleted
I20250112 02:57:28.372767 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:28.373836 22376 ts_tablet_manager.cc:1905] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:28.375001 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 33f89398c7aa4f2a938a476db897c833 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]) successfully deleted
W20250112 02:57:28.375517 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 2601daad8ba742ec88ccfe603440e8e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2601daad8ba742ec88ccfe603440e8e7 already in progress: creating tablet
I20250112 02:57:28.376499 22145 ts_tablet_manager.cc:1428] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250112 02:57:28.377122 22143 raft_consensus.cc:738] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.378051 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]) successfully deleted
I20250112 02:57:28.378681 22581 raft_consensus.cc:491] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.378793 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994e1d114e5a4de5aa1d7651907d718e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.379069 22143 consensus_queue.cc:260] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.379174 22581 raft_consensus.cc:513] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.381187 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:28.381361 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 177) took 1054 ms. Trace:
I20250112 02:57:28.381855 22581 leader_election.cc:290] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:28.381671 22375 rpcz_store.cc:276] 0112 02:57:27.326685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:27.326784 (+ 99us) service_pool.cc:225] Handling call
0112 02:57:27.497319 (+170535us) ts_tablet_manager.cc:1145] Deleting tablet 33f89398c7aa4f2a938a476db897c833
0112 02:57:27.497359 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:28.044462 (+547103us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.236614 (+192152us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.381308 (+144694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":127056}
I20250112 02:57:28.382651 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:28.383700 21980 leader_election.cc:343] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.388901 22145 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.401924 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 33f89398c7aa4f2a938a476db897c833 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]) successfully deleted
I20250112 02:57:28.412547 22143 ts_tablet_manager.cc:1428] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250112 02:57:28.413420 22143 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:28.422185 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:28.423676 21392 catalog_manager.cc:5526] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:28.425103 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:28.425771 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:28.426815 21392 catalog_manager.cc:5526] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
W20250112 02:57:28.430456 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 65eb8c99917247edbdbf5de42f3b505c consensus state has no leader
I20250112 02:57:28.433720 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:28.435657 22376 ts_tablet_manager.cc:1918] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.435990 22376 log.cc:1198] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/36495fb158a440309eddedd3910a7ad3
I20250112 02:57:28.436569 22376 ts_tablet_manager.cc:1939] T 36495fb158a440309eddedd3910a7ad3 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:28.441977 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 2601daad8ba742ec88ccfe603440e8e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2601daad8ba742ec88ccfe603440e8e7 already in progress: creating tablet
I20250112 02:57:28.444839 22143 tablet_bootstrap.cc:654] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.444959 22375 tablet_replica.cc:331] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:28.442189 21392 catalog_manager.cc:5526] T f08c1affbaf247059ab65222e2e95bab P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:28.445720 22375 raft_consensus.cc:2238] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:28.446170 22375 raft_consensus.cc:2267] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:28.446429 22145 tablet_bootstrap.cc:654] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.447806 22375 ts_tablet_manager.cc:1905] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:28.449548 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_5ac32f685843445fb0c4c65fc92dd2ca [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 1 got 0
W20250112 02:57:28.456624 21978 leader_election.cc:343] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.457011 21978 leader_election.cc:304] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.457885 22782 raft_consensus.cc:2743] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.459801 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]) successfully deleted
I20250112 02:57:28.447687 22645 raft_consensus.cc:491] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.462757 22645 raft_consensus.cc:513] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.465157 22645 leader_election.cc:290] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.449676 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:28.450768 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:28.466820 21751 raft_consensus.cc:2463] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:28.468362 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:28.468920 22040 raft_consensus.cc:2463] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:28.469919 21690 leader_election.cc:304] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.475353 22645 raft_consensus.cc:2798] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:28.475703 22645 raft_consensus.cc:491] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:28.476047 22645 raft_consensus.cc:3054] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.481439 22145 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.481834 22145 ts_tablet_manager.cc:1397] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.000s
I20250112 02:57:28.484086 22145 raft_consensus.cc:357] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.484680 22145 raft_consensus.cc:383] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.484930 22145 raft_consensus.cc:738] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:21.247648 22509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:21.258535 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:21.203503 22265 server_base.cc:1034] running on GCE node
I20250112 02:57:28.494961 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:28.496119 22143 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:28.496485 22143 ts_tablet_manager.cc:1397] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.000s
I20250112 02:57:28.499295 22143 raft_consensus.cc:357] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.499948 22143 raft_consensus.cc:383] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.500187 22143 raft_consensus.cc:738] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.503799 22265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:28.507400 22265 hybrid_clock.cc:648] HybridClock initialized: now 1736650648507351 us; error 49 us; skew 500 ppm
I20250112 02:57:28.508668 22265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:28.510607 22144 tablet_bootstrap.cc:492] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:28.500867 22145 consensus_queue.cc:260] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.511085 22144 ts_tablet_manager.cc:1397] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.377s user 0.005s sys 0.004s
I20250112 02:57:28.515234 22144 raft_consensus.cc:357] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.516472 22645 raft_consensus.cc:513] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.519038 22145 ts_tablet_manager.cc:1428] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250112 02:57:28.519881 22145 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.523578 22143 consensus_queue.cc:260] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.525877 22143 ts_tablet_manager.cc:1428] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
W20250112 02:57:28.540305 21691 leader_election.cc:343] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:28.541865 21691 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.542274 21691 leader_election.cc:304] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:28.548362 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:28.548950 22040 raft_consensus.cc:3054] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.557583 22672 raft_consensus.cc:2743] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.559655 22040 raft_consensus.cc:2463] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:28.560565 21690 leader_election.cc:304] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.561385 22672 raft_consensus.cc:2798] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:28.561743 22672 raft_consensus.cc:695] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:28.562371 22672 consensus_queue.cc:237] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.551772 22769 raft_consensus.cc:491] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.568512 22769 raft_consensus.cc:513] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.560889 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.573107 22145 tablet_bootstrap.cc:654] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:28.573856 21835 leader_election.cc:343] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:28.574362 22769 leader_election.cc:290] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.574982 22144 raft_consensus.cc:383] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.547219 22645 leader_election.cc:290] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.555747 22143 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:28.576694 22730 raft_consensus.cc:491] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.577090 22730 raft_consensus.cc:513] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.579533 22730 leader_election.cc:290] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:28.580090 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.582787 22144 raft_consensus.cc:738] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
W20250112 02:57:28.583513 21980 leader_election.cc:343] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.583398 22144 consensus_queue.cc:260] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.590569 22143 tablet_bootstrap.cc:654] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:28.595620 21978 leader_election.cc:343] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.595960 21978 leader_election.cc:304] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.597203 21978 leader_election.cc:304] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.598013 22730 raft_consensus.cc:2743] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.598482 22730 raft_consensus.cc:2758] T e21a0e4b5ebc426383cf332f46492b80 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250112 02:57:28.600481 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21a0e4b5ebc426383cf332f46492b80" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:28.603906 22769 consensus_queue.cc:1035] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:57:28.611027 21836 leader_election.cc:343] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:28.611409 21836 leader_election.cc:304] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:28.613334 22769 raft_consensus.cc:2743] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:28.618490 22144 ts_tablet_manager.cc:1428] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.107s user 0.004s sys 0.000s
I20250112 02:57:28.618662 22524 raft_consensus.cc:491] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.619096 22524 raft_consensus.cc:513] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.620676 22524 leader_election.cc:290] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:28.625398 21836 leader_election.cc:343] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767
W20250112 02:57:28.626325 21835 leader_election.cc:343] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:28.626662 21835 leader_election.cc:304] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:28.627548 22770 consensus_queue.cc:1035] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.627907 22265 webserver.cc:458] Webserver started at http://127.20.217.189:45653/ using document root <none> and password file <none>
I20250112 02:57:28.628628 22770 raft_consensus.cc:2743] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.629282 22265 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:28.629578 22265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:28.646595 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:28.648464 21978 leader_election.cc:343] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.648828 21978 leader_election.cc:304] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.649688 22730 raft_consensus.cc:2743] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:28.650604 21691 leader_election.cc:343] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:28.652009 21691 leader_election.cc:343] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.652954 22144 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:28.659379 22265 fs_manager.cc:714] Time spent opening directory manager: real 0.023s user 0.007s sys 0.001s
I20250112 02:57:28.664186 22645 raft_consensus.cc:491] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.664630 22645 raft_consensus.cc:513] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.671279 22645 leader_election.cc:290] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.672034 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:28.672637 22040 raft_consensus.cc:2463] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:28.673547 21690 leader_election.cc:304] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.674341 22835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:28.677722 22265 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.007s sys 0.001s
I20250112 02:57:28.678128 22265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
format_stamp: "Formatted at 2025-01-12 02:57:02 on dist-test-slave-gppw"
I20250112 02:57:28.681965 22265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:28.683187 22645 raft_consensus.cc:2798] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:28.683534 22645 raft_consensus.cc:491] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:28.683853 22645 raft_consensus.cc:3054] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.686859 22143 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:28.687223 22143 ts_tablet_manager.cc:1397] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.161s user 0.004s sys 0.006s
I20250112 02:57:28.691473 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:28.692811 21691 leader_election.cc:343] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.699855 22145 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.700325 22145 ts_tablet_manager.cc:1397] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.181s user 0.010s sys 0.002s
I20250112 02:57:28.702591 22145 raft_consensus.cc:357] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.703271 22145 raft_consensus.cc:383] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.703518 22145 raft_consensus.cc:738] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.704001 22143 raft_consensus.cc:357] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.704079 22145 consensus_queue.cc:260] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.704643 22143 raft_consensus.cc:383] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.704892 22143 raft_consensus.cc:738] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.707476 22143 consensus_queue.cc:260] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.712245 22730 raft_consensus.cc:491] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.712618 22730 raft_consensus.cc:513] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.715806 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:28.716977 21980 leader_election.cc:343] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.714406 22730 leader_election.cc:290] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:28.718401 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:28.719524 21978 leader_election.cc:343] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.719862 21978 leader_election.cc:304] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.720746 22730 raft_consensus.cc:2743] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:28.730206 22144 tablet_bootstrap.cc:654] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.731386 22143 ts_tablet_manager.cc:1428] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.044s user 0.004s sys 0.002s
I20250112 02:57:28.753612 22145 ts_tablet_manager.cc:1428] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20250112 02:57:28.753175 22672 raft_consensus.cc:491] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:28.756100 22672 raft_consensus.cc:513] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.726917 22645 raft_consensus.cc:513] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.759697 22145 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.759554 22672 leader_election.cc:290] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.761209 22645 leader_election.cc:290] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.762550 22375 ts_tablet_manager.cc:1918] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:28.764375 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:28.764945 22045 raft_consensus.cc:3054] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.768945 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:28.769523 22040 raft_consensus.cc:2463] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:28.770638 21690 leader_election.cc:304] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.770807 22375 log.cc:1198] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/36495fb158a440309eddedd3910a7ad3
I20250112 02:57:28.771479 22672 raft_consensus.cc:2798] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:28.771889 22672 raft_consensus.cc:491] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:28.772241 22672 raft_consensus.cc:3054] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.779196 22672 raft_consensus.cc:513] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.779580 22045 raft_consensus.cc:2463] T d6ca275360ec41ceb0f7ef9bfaedf647 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:28.780570 21690 leader_election.cc:304] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:28.781163 22672 leader_election.cc:290] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:28.781692 22375 ts_tablet_manager.cc:1939] T 36495fb158a440309eddedd3910a7ad3 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:28.784039 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:28.784605 22040 raft_consensus.cc:3054] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:28.782227 22644 raft_consensus.cc:2798] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:28.790349 22644 raft_consensus.cc:695] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:28.791070 22644 consensus_queue.cc:237] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.799451 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:28.783569 22144 tablet_bootstrap.cc:492] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:28.793160 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:28.802716 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:28.803261 22144 ts_tablet_manager.cc:1397] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.150s user 0.008s sys 0.002s
I20250112 02:57:28.805193 22144 raft_consensus.cc:357] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:28.805759 22144 raft_consensus.cc:383] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.805999 22144 raft_consensus.cc:738] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.808276 22143 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:28.811534 21980 consensus_peers.cc:487] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:28.815761 22730 consensus_queue.cc:1035] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:28.816931 22375 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 188) took 1173 ms. Trace:
I20250112 02:57:28.817231 22375 rpcz_store.cc:276] 0112 02:57:27.643140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:27.643222 (+ 82us) service_pool.cc:225] Handling call
0112 02:57:28.444817 (+801595us) ts_tablet_manager.cc:1145] Deleting tablet 36495fb158a440309eddedd3910a7ad3
0112 02:57:28.444858 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:28.694820 (+249962us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.762318 (+ 67498us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:28.816868 (+ 54550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":776269}
I20250112 02:57:28.818353 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]) successfully deleted
W20250112 02:57:28.824448 21691 leader_election.cc:343] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:28.825522 21691 leader_election.cc:343] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:28.826239 21691 leader_election.cc:343] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:28.838182 22143 tablet_bootstrap.cc:654] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.861631 22040 raft_consensus.cc:2463] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:28.862700 21690 leader_election.cc:304] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:28.863595 22672 raft_consensus.cc:2798] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:28.863979 22672 raft_consensus.cc:695] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:28.864663 22672 consensus_queue.cc:237] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.870062 22144 consensus_queue.cc:260] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
W20250112 02:57:28.892274 21379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250112 02:57:28.894203 22144 ts_tablet_manager.cc:1428] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.091s user 0.004s sys 0.001s
I20250112 02:57:28.898026 22144 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:28.903023 22145 tablet_bootstrap.cc:700] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent opening tablet: real 0.133s user 0.004s sys 0.000s
I20250112 02:57:28.903697 22145 tablet_bootstrap.cc:654] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:28.908547 22265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:28.910452 22265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:28.910979 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:28.925452 22143 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:28.928128 22143 ts_tablet_manager.cc:1397] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.002s
I20250112 02:57:28.938050 22145 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:28.938485 22145 ts_tablet_manager.cc:1397] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.179s user 0.008s sys 0.002s
I20250112 02:57:28.940918 22145 raft_consensus.cc:357] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.944573 22145 raft_consensus.cc:383] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.944897 22145 raft_consensus.cc:738] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.930342 22143 raft_consensus.cc:357] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.945827 22143 raft_consensus.cc:383] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:28.946105 22143 raft_consensus.cc:738] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:28.946753 22143 consensus_queue.cc:260] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.945566 22145 consensus_queue.cc:260] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:28.963656 22145 ts_tablet_manager.cc:1428] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20250112 02:57:28.966991 22143 ts_tablet_manager.cc:1428] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250112 02:57:28.967785 22143 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:28.980088 22145 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:28.985222 22145 tablet_bootstrap.cc:654] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.003126 22144 tablet_bootstrap.cc:700] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent opening tablet: real 0.101s user 0.004s sys 0.000s
I20250112 02:57:29.004096 22144 tablet_bootstrap.cc:654] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.018684 22143 tablet_bootstrap.cc:654] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.033890 22855 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:29.001566 22730 raft_consensus.cc:491] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.037564 22730 raft_consensus.cc:513] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.041253 22730 leader_election.cc:290] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.043412 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.044189 22143 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:29.044561 22143 ts_tablet_manager.cc:1397] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.004s
W20250112 02:57:29.044858 21980 leader_election.cc:343] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.045627 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:29.046715 21978 leader_election.cc:343] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.047044 21978 leader_election.cc:304] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.048287 22730 raft_consensus.cc:2743] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.060199 22143 raft_consensus.cc:357] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.060830 22143 raft_consensus.cc:383] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:29.061265 21691 consensus_peers.cc:487] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:29.063386 22145 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:29.061082 22143 raft_consensus.cc:738] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.063794 22145 ts_tablet_manager.cc:1397] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.004s
I20250112 02:57:29.064272 22143 consensus_queue.cc:260] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.066037 22145 raft_consensus.cc:357] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.066705 22145 raft_consensus.cc:383] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.066948 22145 raft_consensus.cc:738] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.068159 22855 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:29.068250 22145 consensus_queue.cc:260] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.070320 22144 tablet_bootstrap.cc:492] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:29.072006 22144 ts_tablet_manager.cc:1397] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.174s user 0.014s sys 0.000s
I20250112 02:57:29.074137 22143 ts_tablet_manager.cc:1428] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.029s user 0.006s sys 0.001s
I20250112 02:57:29.078794 22145 ts_tablet_manager.cc:1428] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20250112 02:57:29.082095 22145 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:29.082562 22144 raft_consensus.cc:357] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.083508 22144 raft_consensus.cc:383] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.083806 22144 raft_consensus.cc:738] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.084507 22144 consensus_queue.cc:260] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.085604 22730 raft_consensus.cc:491] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.086086 22730 raft_consensus.cc:513] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.091585 22145 tablet_bootstrap.cc:654] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.092031 22730 leader_election.cc:290] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:29.092008 22855 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:29.093418 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 304 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:29.094349 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.094696 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:29.095552 21980 leader_election.cc:343] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:29.096654 21978 leader_election.cc:343] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.096968 21978 leader_election.cc:304] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.097759 22730 raft_consensus.cc:2743] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:29.116083 21533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:29.118219 22144 ts_tablet_manager.cc:1428] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20250112 02:57:29.122089 22145 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:29.122483 22145 ts_tablet_manager.cc:1397] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.005s
W20250112 02:57:29.122735 21533 leader_election.cc:336] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049): Network error: Client connection negotiation failed: client connection to 127.20.217.189:36049: connect: Connection refused (error 111)
I20250112 02:57:29.124385 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b; no voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:29.126029 22855 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:29.127707 22145 raft_consensus.cc:357] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.128319 22145 raft_consensus.cc:383] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.128576 22145 raft_consensus.cc:738] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.129716 22144 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:29.131111 22145 consensus_queue.cc:260] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.137609 22145 ts_tablet_manager.cc:1428] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250112 02:57:29.140816 22143 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:29.151136 22145 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:29.163918 22143 tablet_bootstrap.cc:654] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.167670 22145 tablet_bootstrap.cc:654] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.195195 22144 tablet_bootstrap.cc:654] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.199270 22143 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:29.199688 22143 ts_tablet_manager.cc:1397] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.008s
I20250112 02:57:29.204326 22143 raft_consensus.cc:357] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.204934 22143 raft_consensus.cc:383] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.205184 22143 raft_consensus.cc:738] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.207860 22145 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:29.208425 22145 ts_tablet_manager.cc:1397] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.000s
I20250112 02:57:29.210829 22145 raft_consensus.cc:357] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.211524 22145 raft_consensus.cc:383] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.211766 22145 raft_consensus.cc:738] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.225239 22144 tablet_bootstrap.cc:492] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:29.225627 22144 ts_tablet_manager.cc:1397] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.000s
I20250112 02:57:29.226802 22145 consensus_queue.cc:260] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.227906 22144 raft_consensus.cc:357] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.236892 22145 ts_tablet_manager.cc:1428] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250112 02:57:29.237008 21731 tablet_service.cc:1467] Processing CreateTablet for tablet a345af15e7664d01a184193ddb51572b (DEFAULT_TABLE table=default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.238081 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a345af15e7664d01a184193ddb51572b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.239777 22145 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:29.240108 21882 tablet_service.cc:1467] Processing CreateTablet for tablet a345af15e7664d01a184193ddb51572b (DEFAULT_TABLE table=default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.241248 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a345af15e7664d01a184193ddb51572b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.249339 22015 tablet_service.cc:1467] Processing CreateTablet for tablet a345af15e7664d01a184193ddb51572b (DEFAULT_TABLE table=default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.250687 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a345af15e7664d01a184193ddb51572b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.254573 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 334ed386f72d4fca870ade9d6a44589a (DEFAULT_TABLE table=default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.254531 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 334ed386f72d4fca870ade9d6a44589a (DEFAULT_TABLE table=default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.255846 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334ed386f72d4fca870ade9d6a44589a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.255961 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334ed386f72d4fca870ade9d6a44589a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.262224 22145 tablet_bootstrap.cc:654] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.264966 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 68f5144c9c644a7f8ca95082459c4556 (DEFAULT_TABLE table=default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.265705 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 334ed386f72d4fca870ade9d6a44589a (DEFAULT_TABLE table=default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.266016 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5144c9c644a7f8ca95082459c4556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.266708 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 334ed386f72d4fca870ade9d6a44589a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.270681 21902 raft_consensus.cc:3054] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.272264 22145 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:29.272763 22145 ts_tablet_manager.cc:1397] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.000s
I20250112 02:57:29.277312 22730 raft_consensus.cc:491] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.228514 22144 raft_consensus.cc:383] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.277773 22730 raft_consensus.cc:513] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.278194 22144 raft_consensus.cc:738] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.279013 22144 consensus_queue.cc:260] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.279100 22145 raft_consensus.cc:357] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.279942 22145 raft_consensus.cc:383] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.280236 22145 raft_consensus.cc:738] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.280403 22730 leader_election.cc:290] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.280525 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:29.280898 22145 consensus_queue.cc:260] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:29.281858 21978 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.282732 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:29.284759 21980 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.285102 21980 leader_election.cc:304] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.288916 22581 raft_consensus.cc:2743] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.293543 22581 consensus_queue.cc:1035] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.298470 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 68f5144c9c644a7f8ca95082459c4556 (DEFAULT_TABLE table=default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.299589 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5144c9c644a7f8ca95082459c4556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.300467 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 68f5144c9c644a7f8ca95082459c4556 (DEFAULT_TABLE table=default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.300626 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 (DEFAULT_TABLE table=default.Table_7d88c7e3065640629e9d17b64174c9e4 [id=f1b12e4291bd474983056b14b9a070f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.301416 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f5144c9c644a7f8ca95082459c4556. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.301579 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922b22f0b1284bc4a732371a61cb1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.309669 22145 ts_tablet_manager.cc:1428] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20250112 02:57:29.310530 22145 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:29.232679 22143 consensus_queue.cc:260] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.332783 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 (DEFAULT_TABLE table=default.Table_7d88c7e3065640629e9d17b64174c9e4 [id=f1b12e4291bd474983056b14b9a070f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.333835 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922b22f0b1284bc4a732371a61cb1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.333712 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 (DEFAULT_TABLE table=default.Table_7d88c7e3065640629e9d17b64174c9e4 [id=f1b12e4291bd474983056b14b9a070f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.334470 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9b3e6c84484575acd46e61dd1ea28a (DEFAULT_TABLE table=default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.334751 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 922b22f0b1284bc4a732371a61cb1ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.335413 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9b3e6c84484575acd46e61dd1ea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.344456 22782 consensus_queue.cc:1035] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.357538 22782 raft_consensus.cc:491] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.357919 22782 raft_consensus.cc:513] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.359848 22782 leader_election.cc:290] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.360684 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:29.362105 21980 leader_election.cc:343] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.362341 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:29.363940 21978 leader_election.cc:343] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.364271 21978 leader_election.cc:304] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.367866 22143 ts_tablet_manager.cc:1428] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.168s user 0.006s sys 0.001s
I20250112 02:57:29.376135 22645 raft_consensus.cc:491] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.376605 22645 raft_consensus.cc:513] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.382236 22645 leader_election.cc:290] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.393335 21876 tablet_service.cc:1467] Processing CreateTablet for tablet e7d8db14a8054ae69daf42dd76f35511 (DEFAULT_TABLE table=default.Table_d498875b28e14a3d815a24d80cb3cc7b [id=e1ba72633675415b8edec82d382ca718]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.394520 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d8db14a8054ae69daf42dd76f35511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.395823 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (DEFAULT_TABLE table=default.Table_ff23deca1f7543abbaab1dcbea953bab [id=023cccda18f7466c8de7d483f8055740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.396750 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750246f0a0c549229d4e7a87cd6ddfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.398864 22143 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:29.412191 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9b3e6c84484575acd46e61dd1ea28a (DEFAULT_TABLE table=default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.413116 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9b3e6c84484575acd46e61dd1ea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.415231 21726 tablet_service.cc:1467] Processing CreateTablet for tablet e7d8db14a8054ae69daf42dd76f35511 (DEFAULT_TABLE table=default.Table_d498875b28e14a3d815a24d80cb3cc7b [id=e1ba72633675415b8edec82d382ca718]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.416244 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d8db14a8054ae69daf42dd76f35511. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:29.440083 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a69123c4ab384e0380c62e52b9f3b42d consensus state has no leader
I20250112 02:57:29.443210 22145 tablet_bootstrap.cc:654] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.445469 22143 tablet_bootstrap.cc:654] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.457278 22144 ts_tablet_manager.cc:1428] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.231s user 0.009s sys 0.000s
I20250112 02:57:29.467110 22006 tablet_service.cc:1467] Processing CreateTablet for tablet e7d8db14a8054ae69daf42dd76f35511 (DEFAULT_TABLE table=default.Table_d498875b28e14a3d815a24d80cb3cc7b [id=e1ba72633675415b8edec82d382ca718]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.441676 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 7c9b3e6c84484575acd46e61dd1ea28a (DEFAULT_TABLE table=default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.468868 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_5ac32f685843445fb0c4c65fc92dd2ca [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 1 got 0
I20250112 02:57:29.469879 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:29.470633 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:29.473275 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.473817 22041 raft_consensus.cc:2463] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:29.482707 21690 leader_election.cc:304] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:29.485880 22144 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:29.491559 22645 consensus_queue.cc:1035] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.494479 22644 raft_consensus.cc:2798] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:29.494800 22644 raft_consensus.cc:491] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:29.495077 22644 raft_consensus.cc:3054] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.501421 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (DEFAULT_TABLE table=default.Table_ff23deca1f7543abbaab1dcbea953bab [id=023cccda18f7466c8de7d483f8055740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.501595 22644 raft_consensus.cc:513] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.501791 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.503233 22644 leader_election.cc:290] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.503907 22903 raft_consensus.cc:491] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.504287 22903 raft_consensus.cc:513] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:29.505607 21691 leader_election.cc:343] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.505899 22903 leader_election.cc:290] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.507068 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:29.509943 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.510859 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.511463 22045 raft_consensus.cc:2463] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:29.471221 21387 catalog_manager.cc:5526] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:29.512377 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:29.512696 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750246f0a0c549229d4e7a87cd6ddfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.512871 22040 raft_consensus.cc:3054] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.513028 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:29.513634 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:29.517547 22019 tablet_service.cc:1467] Processing CreateTablet for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (DEFAULT_TABLE table=default.Table_ff23deca1f7543abbaab1dcbea953bab [id=023cccda18f7466c8de7d483f8055740]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.517652 21690 leader_election.cc:304] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:29.518580 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 750246f0a0c549229d4e7a87cd6ddfd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.518857 22644 raft_consensus.cc:2798] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:29.519163 22644 raft_consensus.cc:491] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:29.519474 22644 raft_consensus.cc:3054] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.527240 22730 raft_consensus.cc:491] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.527707 22730 raft_consensus.cc:513] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.529490 22730 leader_election.cc:290] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.530256 22040 raft_consensus.cc:2463] T f663e183c2ed49b98327b2ad0de12ed9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:29.531100 21690 leader_election.cc:304] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:29.531718 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.531916 22645 raft_consensus.cc:2798] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:29.532341 22645 raft_consensus.cc:695] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:29.533075 22645 consensus_queue.cc:237] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:29.533943 21980 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:29.535316 21377 catalog_manager.cc:4945] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): alter failed for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]),no further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
I20250112 02:57:29.548303 22006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7d8db14a8054ae69daf42dd76f35511. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.556499 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:29.561877 22143 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:29.562304 22143 ts_tablet_manager.cc:1397] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.164s user 0.004s sys 0.007s
I20250112 02:57:29.563185 22782 raft_consensus.cc:2743] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.564528 22143 raft_consensus.cc:357] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.565200 22143 raft_consensus.cc:383] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.565460 22143 raft_consensus.cc:738] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.570500 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf (DEFAULT_TABLE table=default.Table_e86223dc27604b0488bcaf442de062a1 [id=5177daf22b3a46af8543a3acbaa05c28]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:29.572357 21691 leader_election.cc:343] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.576004 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.587122 22145 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:29.587486 22145 ts_tablet_manager.cc:1397] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.277s user 0.015s sys 0.004s
I20250112 02:57:29.581092 22644 raft_consensus.cc:513] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.591368 22644 leader_election.cc:290] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.566079 22143 consensus_queue.cc:260] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.592546 22672 raft_consensus.cc:491] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.592942 22672 raft_consensus.cc:513] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.596115 22672 leader_election.cc:290] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.597671 22143 ts_tablet_manager.cc:1428] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250112 02:57:29.598439 22143 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:29.581400 21691 leader_election.cc:343] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:29.605172 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1800d0eb7a9a48d295aca16607291add consensus state has no leader
I20250112 02:57:29.605727 21391 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 259) took 1514 ms. Trace:
I20250112 02:57:29.609257 22145 raft_consensus.cc:357] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.606057 21391 rpcz_store.cc:276] 0112 02:57:28.091292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:28.092832 (+ 1540us) service_pool.cc:225] Handling call
0112 02:57:28.093712 (+ 880us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:28.094349 (+ 637us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:28.094855 (+ 506us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:28.095004 (+ 149us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:28.109599 (+ 14595us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:29.605660 (+1496061us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:28.096086 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:28.096413 (+ 327us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:28.096436 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:28.096448 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:28.096678 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:28.096715 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:28.096735 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:28.096864 (+ 129us) tablet.cc:698] Row locks acquired
0112 02:57:28.096875 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:28.096977 (+ 102us) write_op.cc:270] Start()
0112 02:57:28.097043 (+ 66us) write_op.cc:276] Timestamp: P: 1736650648096904 usec, L: 0
0112 02:57:28.097057 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:28.097439 (+ 382us) log.cc:844] Serialized 405 byte log entry
0112 02:57:28.105405 (+ 7966us) op_driver.cc:464] REPLICATION: finished
0112 02:57:28.105629 (+ 224us) write_op.cc:301] APPLY: starting
0112 02:57:28.105675 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:28.105752 (+ 77us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:28.105831 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:28.105984 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:28.106095 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:28.106143 (+ 48us) write_op.cc:312] APPLY: finished
0112 02:57:28.106292 (+ 149us) log.cc:844] Serialized 23 byte log entry
0112 02:57:28.106879 (+ 587us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.106930 (+ 51us) write_op.cc:454] Released schema lock
0112 02:57:28.107090 (+ 160us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:28.107274 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.107300 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":1070,"apply.run_wall_time_us":1618,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1580,"prepare.run_wall_time_us":2393,"raft.queue_time_us":150,"raft.run_cpu_time_us":440,"raft.run_wall_time_us":1078,"replication_time_us":8275}]]}
I20250112 02:57:29.609894 22145 raft_consensus.cc:383] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.610121 22145 raft_consensus.cc:738] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.621774 22145 consensus_queue.cc:260] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:29.629025 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 47671.4us
W20250112 02:57:29.629747 21691 consensus_peers.cc:487] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:29.630525 21978 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.630657 22782 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.630887 21978 leader_election.cc:304] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.631017 22782 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.631763 22730 raft_consensus.cc:2743] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.632833 22782 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.634081 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:29.635597 21980 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.639911 22144 tablet_bootstrap.cc:700] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Time spent opening tablet: real 0.150s user 0.004s sys 0.000s
I20250112 02:57:29.641932 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf (DEFAULT_TABLE table=default.Table_e86223dc27604b0488bcaf442de062a1 [id=5177daf22b3a46af8543a3acbaa05c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.652949 22145 ts_tablet_manager.cc:1428] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.065s user 0.001s sys 0.004s
I20250112 02:57:29.603186 22916 tablet.cc:1721] T e0977a10252840a381234f02bb7339b3 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:29.673362 22144 tablet_bootstrap.cc:654] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.669222 21877 tablet_service.cc:1467] Processing CreateTablet for tablet edfa495bfefc46b998c2f870d9014900 (DEFAULT_TABLE table=default.Table_c562ca46d2f349e58930eef499aef1cf [id=f7e10ced6a8647e8b15c3ca704c1fdb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.681684 22143 tablet_bootstrap.cc:654] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:29.682431 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfa495bfefc46b998c2f870d9014900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.682605 22008 tablet_service.cc:1467] Processing CreateTablet for tablet edfa495bfefc46b998c2f870d9014900 (DEFAULT_TABLE table=default.Table_c562ca46d2f349e58930eef499aef1cf [id=f7e10ced6a8647e8b15c3ca704c1fdb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.699769 22900 raft_consensus.cc:491] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.700248 22900 raft_consensus.cc:513] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.702890 22900 leader_election.cc:290] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.707242 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:29.713004 22143 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:29.713390 22143 ts_tablet_manager.cc:1397] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.115s user 0.004s sys 0.004s
I20250112 02:57:29.715396 22143 raft_consensus.cc:357] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.716006 22143 raft_consensus.cc:383] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.716243 22143 raft_consensus.cc:738] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.724139 21899 raft_consensus.cc:3054] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.726776 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf (DEFAULT_TABLE table=default.Table_e86223dc27604b0488bcaf442de062a1 [id=5177daf22b3a46af8543a3acbaa05c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.727749 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.731467 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:29.731380 21726 tablet_service.cc:1467] Processing CreateTablet for tablet edfa495bfefc46b998c2f870d9014900 (DEFAULT_TABLE table=default.Table_c562ca46d2f349e58930eef499aef1cf [id=f7e10ced6a8647e8b15c3ca704c1fdb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.732643 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfa495bfefc46b998c2f870d9014900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.737351 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1144ac759a4db496f67cdc9ec35020 (DEFAULT_TABLE table=default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.738226 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1144ac759a4db496f67cdc9ec35020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.739176 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.743432 22920 tablet.cc:1721] T e0977a10252840a381234f02bb7339b3 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:29.756345 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:29.760159 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.760542 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.760740 22045 raft_consensus.cc:2463] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:29.761894 21690 leader_election.cc:304] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:29.761986 21384 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:29.761962 21836 leader_election.cc:304] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:29.762770 21384 catalog_manager.cc:5526] T 1f21e3c769364b58bd371aac338db1f2 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:29.763440 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:29.763931 22041 raft_consensus.cc:3054] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.764432 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:29.765717 21388 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 228) took 3478 ms. Trace:
I20250112 02:57:29.773360 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1144ac759a4db496f67cdc9ec35020 (DEFAULT_TABLE table=default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.774407 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1144ac759a4db496f67cdc9ec35020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.775200 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:29.775439 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c9b3e6c84484575acd46e61dd1ea28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.777098 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 779badaa22e6404aaaf1818e86228bd4 (DEFAULT_TABLE table=default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.778028 22011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779badaa22e6404aaaf1818e86228bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.778020 22144 tablet_bootstrap.cc:492] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:29.782526 21874 tablet_service.cc:1467] Processing CreateTablet for tablet d94cda34ff2e44cfa543362a007f1400 (DEFAULT_TABLE table=default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.783507 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94cda34ff2e44cfa543362a007f1400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.745123 22143 consensus_queue.cc:260] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.766000 21388 rpcz_store.cc:276] 0112 02:57:26.286845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.287263 (+ 418us) service_pool.cc:225] Handling call
0112 02:57:26.311869 (+ 24606us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:26.312387 (+ 518us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:26.312490 (+ 103us) catalog_manager.cc:2762] Locking tablets
0112 02:57:27.265592 (+953102us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:27.556049 (+290457us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:27.556224 (+ 175us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:29.764350 (+2208126us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:29.765661 (+ 1311us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:27.335826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:27.336124 (+ 298us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:27.336144 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:27.336156 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:27.336420 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:27.336449 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:27.336469 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:27.336698 (+ 229us) tablet.cc:698] Row locks acquired
0112 02:57:27.336710 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:27.336795 (+ 85us) write_op.cc:270] Start()
0112 02:57:27.336879 (+ 84us) write_op.cc:276] Timestamp: P: 1736650647336891 usec, L: 0
0112 02:57:27.336894 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:27.337273 (+ 379us) log.cc:844] Serialized 734 byte log entry
0112 02:57:27.549159 (+211886us) op_driver.cc:464] REPLICATION: finished
0112 02:57:27.550072 (+ 913us) write_op.cc:301] APPLY: starting
0112 02:57:27.550121 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:27.550207 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:27.550318 (+ 111us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:27.550550 (+ 232us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:27.550673 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:27.550713 (+ 40us) write_op.cc:312] APPLY: finished
0112 02:57:27.550865 (+ 152us) log.cc:844] Serialized 29 byte log entry
0112 02:57:27.550976 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:27.555523 (+ 4547us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b6000020e78. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed9cda81 00007f7eedb0e9ed 00007f7eedb0e8a5 00007f7eedab05a9 00007f7eedaaf725 00007f7eedaaf11c 00007f7eeda9b1b9 00007f7eeda9a983 00007f7eeda9eb91 00007f7eeda9eb49 00007f7eeda9ead9
0112 02:57:27.555555 (+ 32us) write_op.cc:454] Released schema lock
0112 02:57:27.555583 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:27.555782 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:27.555810 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":771,"apply.run_cpu_time_us":2906,"apply.run_wall_time_us":5698,"num_ops":2,"prepare.queue_time_us":68585,"prepare.run_cpu_time_us":1709,"prepare.run_wall_time_us":43984,"raft.queue_time_us":41515,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":212188,"spinlock_wait_cycles":5132160}]]}
I20250112 02:57:29.778422 22144 ts_tablet_manager.cc:1397] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.293s user 0.009s sys 0.000s
I20250112 02:57:29.793936 22144 raft_consensus.cc:357] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.794598 22144 raft_consensus.cc:383] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:29.794847 22144 raft_consensus.cc:738] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:29.795408 22144 consensus_queue.cc:260] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.800894 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 779badaa22e6404aaaf1818e86228bd4 (DEFAULT_TABLE table=default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.801826 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779badaa22e6404aaaf1818e86228bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.802953 22930 tablet_replica.cc:331] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:29.804031 22930 raft_consensus.cc:2238] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:29.804585 22930 raft_consensus.cc:2267] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:29.806885 22930 ts_tablet_manager.cc:1905] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:29.806835 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:29.808776 22143 ts_tablet_manager.cc:1428] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.095s user 0.007s sys 0.001s
I20250112 02:57:29.809376 22902 raft_consensus.cc:2798] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:29.809731 22902 raft_consensus.cc:491] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:29.810022 22902 raft_consensus.cc:3054] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:29.814597 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 52796.9us
I20250112 02:57:29.815258 22644 raft_consensus.cc:491] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.815654 22644 raft_consensus.cc:513] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.817449 22644 leader_election.cc:290] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.818305 22644 consensus_queue.cc:1035] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.819957 22644 consensus_queue.cc:1035] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.826525 22902 raft_consensus.cc:513] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.827220 21731 tablet_service.cc:1467] Processing CreateTablet for tablet 7a1144ac759a4db496f67cdc9ec35020 (DEFAULT_TABLE table=default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.828235 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:29.828743 21691 leader_election.cc:343] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:29.829365 21691 leader_election.cc:343] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.829559 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 779badaa22e6404aaaf1818e86228bd4 (DEFAULT_TABLE table=default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:29.830083 21691 consensus_peers.cc:487] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:29.830626 21736 tablet_service.cc:1467] Processing CreateTablet for tablet d94cda34ff2e44cfa543362a007f1400 (DEFAULT_TABLE table=default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.830862 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:29.843367 21876 tablet_service.cc:1467] Processing CreateTablet for tablet a08141e3615244a9b266cd197d9b9ef3 (DEFAULT_TABLE table=default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.846427 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:29.849463 21691 leader_election.cc:343] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.855727 22645 consensus_queue.cc:1035] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:29.860102 22769 raft_consensus.cc:2743] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.862025 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779badaa22e6404aaaf1818e86228bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.862819 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94cda34ff2e44cfa543362a007f1400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.863343 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1144ac759a4db496f67cdc9ec35020. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.869482 22144 ts_tablet_manager.cc:1428] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.078s user 0.004s sys 0.000s
W20250112 02:57:29.869827 21691 leader_election.cc:343] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.863507 22933 tablet_replica.cc:331] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:29.879040 22933 raft_consensus.cc:2238] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:29.881042 22933 raft_consensus.cc:2267] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:29.889606 22933 ts_tablet_manager.cc:1905] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:29.899114 21736 tablet_service.cc:1467] Processing CreateTablet for tablet a08141e3615244a9b266cd197d9b9ef3 (DEFAULT_TABLE table=default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.879539 22931 tablet_replica.cc:331] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:29.898823 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08141e3615244a9b266cd197d9b9ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.906524 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08141e3615244a9b266cd197d9b9ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.906549 22931 raft_consensus.cc:2238] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:29.907369 22931 raft_consensus.cc:2267] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:29.914528 22900 raft_consensus.cc:491] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.914983 22900 raft_consensus.cc:513] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:29.917827 22900 leader_election.cc:290] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:29.918483 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:29.919539 22931 ts_tablet_manager.cc:1905] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:29.927071 22144 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:29.931046 22933 ts_tablet_manager.cc:1918] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:29.931531 22933 log.cc:1198] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:29.932981 22933 ts_tablet_manager.cc:1939] T 27f0d6a5afd140a8af77a5a39dc39dd2 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:29.935150 22919 raft_consensus.cc:491] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:29.940078 22919 raft_consensus.cc:513] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:29.940999 21899 raft_consensus.cc:3054] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.942284 22919 leader_election.cc:290] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.952355 22902 leader_election.cc:290] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:29.816505 22019 tablet_service.cc:1467] Processing CreateTablet for tablet d94cda34ff2e44cfa543362a007f1400 (DEFAULT_TABLE table=default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:29.824882 21978 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.962905 21978 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:29.963366 22143 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:29.963593 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 138796us
I20250112 02:57:29.965444 22782 raft_consensus.cc:2743] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:29.829046 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:29.835980 22145 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:29.840029 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.841940 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.975113 22038 raft_consensus.cc:3054] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.977257 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d94cda34ff2e44cfa543362a007f1400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.848875 22012 tablet_service.cc:1467] Processing CreateTablet for tablet a08141e3615244a9b266cd197d9b9ef3 (DEFAULT_TABLE table=default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:29.979218 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a08141e3615244a9b266cd197d9b9ef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.982780 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edfa495bfefc46b998c2f870d9014900. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:29.988018 21902 raft_consensus.cc:3054] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.810309 22041 raft_consensus.cc:2463] T 2cbf5e2a26474c98b9cb48b5bbdceafb P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:29.998154 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:30.001998 21691 consensus_peers.cc:487] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:30.003964 21691 leader_election.cc:343] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:29.880434 22936 tablet.cc:1721] T e0977a10252840a381234f02bb7339b3 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:30.004041 22900 raft_consensus.cc:491] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e0406c08e9714e87bdd7fd6b66c9756d)
I20250112 02:57:30.004866 22900 raft_consensus.cc:513] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:30.009160 22900 leader_election.cc:290] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.009760 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.013374 21690 leader_election.cc:304] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.016362 22902 raft_consensus.cc:2798] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.016542 22143 tablet_bootstrap.cc:654] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.016901 22902 raft_consensus.cc:695] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.017942 22045 raft_consensus.cc:2463] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:30.017714 22902 consensus_queue.cc:237] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.019021 21690 leader_election.cc:304] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.019748 22645 raft_consensus.cc:2798] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:30.020076 22645 raft_consensus.cc:491] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:30.020367 22645 raft_consensus.cc:3054] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.959287 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:29.992988 22145 tablet_bootstrap.cc:654] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.027793 22043 raft_consensus.cc:2463] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:30.028957 21690 leader_election.cc:304] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.029733 22672 raft_consensus.cc:2798] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:30.030140 22672 raft_consensus.cc:491] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:30.030488 22672 raft_consensus.cc:3054] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:29.996024 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1800d0eb7a9a48d295aca16607291add" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.034595 22038 raft_consensus.cc:2463] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:30.036643 22730 raft_consensus.cc:491] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.037071 22730 raft_consensus.cc:513] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.038946 22730 leader_election.cc:290] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.041864 21690 leader_election.cc:304] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.040921 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:30.044205 21980 leader_election.cc:343] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.046244 22145 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.046690 22145 ts_tablet_manager.cc:1397] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.211s user 0.008s sys 0.004s
W20250112 02:57:30.046653 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 81624.3us
I20250112 02:57:30.046788 22919 raft_consensus.cc:2798] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.047243 22919 raft_consensus.cc:695] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.048010 22919 consensus_queue.cc:237] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.048864 22145 raft_consensus.cc:357] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.049484 22145 raft_consensus.cc:383] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.049712 22145 raft_consensus.cc:738] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.051232 21836 leader_election.cc:304] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:30.054121 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c80b7813b50d4739a76776d5d16434f3" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.055352 21836 leader_election.cc:304] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:30.056063 22145 consensus_queue.cc:260] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.057634 22144 tablet_bootstrap.cc:654] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.058571 22145 ts_tablet_manager.cc:1428] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20250112 02:57:30.059329 22145 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:30.064025 22145 tablet_bootstrap.cc:654] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.065555 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.070106 22900 raft_consensus.cc:2743] T c80b7813b50d4739a76776d5d16434f3 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:57:30.071274 21978 leader_election.cc:343] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.071789 21978 leader_election.cc:304] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.073313 22782 raft_consensus.cc:2743] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:30.084057 22672 raft_consensus.cc:513] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.089056 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:30.089711 22038 raft_consensus.cc:3054] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.045915 22645 raft_consensus.cc:513] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.094563 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 08e25cf536254f3ea9d2c1e97a32b85b (DEFAULT_TABLE table=default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.095597 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08e25cf536254f3ea9d2c1e97a32b85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.096167 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (table default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]) successfully deleted
I20250112 02:57:30.097524 22645 leader_election.cc:290] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.099185 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:30.099805 22039 raft_consensus.cc:3054] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.101595 22145 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.103698 22145 ts_tablet_manager.cc:1397] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.016s
I20250112 02:57:30.103986 21876 tablet_service.cc:1467] Processing CreateTablet for tablet eab717cb5d794d15b9b51881b86501fc (DEFAULT_TABLE table=default.Table_706a3ed7905244fbbf03303bdbde0fb9 [id=2216f55937994af499e8c0cdfbf951ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.104910 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab717cb5d794d15b9b51881b86501fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.105741 22145 raft_consensus.cc:357] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.106159 22039 raft_consensus.cc:2463] T 67260685978247898ad224128ed8964f P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:30.107645 21690 leader_election.cc:304] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.106343 22145 raft_consensus.cc:383] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.113442 22145 raft_consensus.cc:738] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.114773 22145 consensus_queue.cc:260] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.117812 22015 tablet_service.cc:1467] Processing CreateTablet for tablet eab717cb5d794d15b9b51881b86501fc (DEFAULT_TABLE table=default.Table_706a3ed7905244fbbf03303bdbde0fb9 [id=2216f55937994af499e8c0cdfbf951ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.118891 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab717cb5d794d15b9b51881b86501fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.119097 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.120110 22145 ts_tablet_manager.cc:1428] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250112 02:57:30.120229 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d2021384bb41b58a66fcb7afaeaca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.122776 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 9c31714d18414800845671d783ac8173 (DEFAULT_TABLE table=default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.123978 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c31714d18414800845671d783ac8173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.124389 22145 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:30.127683 22143 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:30.128130 22143 ts_tablet_manager.cc:1397] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.165s user 0.008s sys 0.002s
I20250112 02:57:30.127650 21391 catalog_manager.cc:5526] T e21a0e4b5ebc426383cf332f46492b80 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:30.131136 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.132086 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d2021384bb41b58a66fcb7afaeaca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.135438 22145 tablet_bootstrap.cc:654] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.138481 22645 raft_consensus.cc:2798] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.142194 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:30.145133 22931 ts_tablet_manager.cc:1918] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:30.145524 22931 log.cc:1198] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:30.147742 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:30.151180 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 08e25cf536254f3ea9d2c1e97a32b85b (DEFAULT_TABLE table=default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.152318 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08e25cf536254f3ea9d2c1e97a32b85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.146376 22931 ts_tablet_manager.cc:1939] T 27f0d6a5afd140a8af77a5a39dc39dd2 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:30.155604 22672 leader_election.cc:290] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.139250 22645 raft_consensus.cc:695] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.156535 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 486dd5218bad42b0bb9064b7a14f3a4e (DEFAULT_TABLE table=default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.156816 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (table default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]) successfully deleted
I20250112 02:57:30.157827 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486dd5218bad42b0bb9064b7a14f3a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.148449 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:30.156690 22645 consensus_queue.cc:237] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.159543 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:30.160691 21391 catalog_manager.cc:5526] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:30.161885 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 2 got 0
I20250112 02:57:30.163440 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:30.165062 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67260685978247898ad224128ed8964f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:30.171244 22143 raft_consensus.cc:357] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.171968 22143 raft_consensus.cc:383] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.172327 22143 raft_consensus.cc:738] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.173033 22143 consensus_queue.cc:260] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.132666 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 486dd5218bad42b0bb9064b7a14f3a4e (DEFAULT_TABLE table=default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.179737 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486dd5218bad42b0bb9064b7a14f3a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.162436 22038 raft_consensus.cc:2463] T 0df734ba87264abd803237a5bbb32da2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:30.193951 22936 tablet.cc:1721] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:30.196952 22920 tablet.cc:1721] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:30.194008 21391 catalog_manager.cc:5526] T d6ca275360ec41ceb0f7ef9bfaedf647 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:30.207085 21690 leader_election.cc:304] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.209225 22900 raft_consensus.cc:491] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.209648 22900 raft_consensus.cc:513] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } }
I20250112 02:57:30.211416 22900 leader_election.cc:290] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.213583 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.217861 22769 raft_consensus.cc:2743] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:30.219432 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 08e25cf536254f3ea9d2c1e97a32b85b (DEFAULT_TABLE table=default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.220448 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08e25cf536254f3ea9d2c1e97a32b85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.227452 21874 tablet_service.cc:1467] Processing CreateTablet for tablet e861bab8dcef41328095954a46e00a96 (DEFAULT_TABLE table=default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.228492 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e861bab8dcef41328095954a46e00a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.230378 22143 ts_tablet_manager.cc:1428] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.102s user 0.004s sys 0.002s
I20250112 02:57:30.231204 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.232164 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.236783 22019 tablet_service.cc:1467] Processing CreateTablet for tablet e861bab8dcef41328095954a46e00a96 (DEFAULT_TABLE table=default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.237825 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e861bab8dcef41328095954a46e00a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.238333 22902 raft_consensus.cc:2798] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.239048 22902 raft_consensus.cc:695] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.239858 22902 consensus_queue.cc:237] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.251302 22930 ts_tablet_manager.cc:1918] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:30.251745 22930 log.cc:1198] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:30.252566 22930 ts_tablet_manager.cc:1939] T 27f0d6a5afd140a8af77a5a39dc39dd2 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:30.253928 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 9c31714d18414800845671d783ac8173 (DEFAULT_TABLE table=default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.254909 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c31714d18414800845671d783ac8173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.256613 22145 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.256971 22145 ts_tablet_manager.cc:1397] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.133s user 0.023s sys 0.025s
I20250112 02:57:30.259135 22145 raft_consensus.cc:357] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.259765 22145 raft_consensus.cc:383] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.260002 22145 raft_consensus.cc:738] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.260536 22145 consensus_queue.cc:260] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.262915 22145 ts_tablet_manager.cc:1428] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250112 02:57:30.263988 22145 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:30.265781 21731 tablet_service.cc:1467] Processing CreateTablet for tablet eab717cb5d794d15b9b51881b86501fc (DEFAULT_TABLE table=default.Table_706a3ed7905244fbbf03303bdbde0fb9 [id=2216f55937994af499e8c0cdfbf951ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.266746 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eab717cb5d794d15b9b51881b86501fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.269631 22143 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:30.265733 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 486dd5218bad42b0bb9064b7a14f3a4e (DEFAULT_TABLE table=default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.271435 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486dd5218bad42b0bb9064b7a14f3a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.274013 22942 raft_consensus.cc:491] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.274446 22942 raft_consensus.cc:513] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.277227 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:30.278337 21980 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.279594 22942 leader_election.cc:290] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.281888 22145 tablet_bootstrap.cc:654] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.286744 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 27f0d6a5afd140a8af77a5a39dc39dd2 (table default.Table_2974589a1bbd43d0b58eac1aeed1c19f [id=0441b893cacf40f99116b52d5c201a1f]) successfully deleted
I20250112 02:57:30.292912 22143 tablet_bootstrap.cc:654] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.299557 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.300418 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03d2021384bb41b58a66fcb7afaeaca8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.303313 21727 tablet_service.cc:1467] Processing CreateTablet for tablet e861bab8dcef41328095954a46e00a96 (DEFAULT_TABLE table=default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.304260 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e861bab8dcef41328095954a46e00a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.305284 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.306350 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.308827 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 9c31714d18414800845671d783ac8173 (DEFAULT_TABLE table=default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.309949 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c31714d18414800845671d783ac8173. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.310082 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf8bf172abad43ac9a519310334d1640" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.311368 21836 leader_election.cc:304] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:30.311555 22144 tablet_bootstrap.cc:492] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:30.311947 22144 ts_tablet_manager.cc:1397] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.385s user 0.010s sys 0.007s
I20250112 02:57:30.312280 22769 raft_consensus.cc:2743] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:30.313238 22950 log.cc:926] Time spent T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250112 02:57:30.314620 22144 raft_consensus.cc:357] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.315405 22144 raft_consensus.cc:383] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.315639 22144 raft_consensus.cc:738] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.324761 22965 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.189:36049 every 8 connection(s)
I20250112 02:57:30.325448 22144 consensus_queue.cc:260] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.329232 22265 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.189:36049
I20250112 02:57:30.333060 22143 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:30.339255 22143 ts_tablet_manager.cc:1397] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.000s
I20250112 02:57:30.345611 22265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
I20250112 02:57:30.346722 22144 ts_tablet_manager.cc:1428] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250112 02:57:30.350137 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 22265
I20250112 02:57:30.350625 22145 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.351055 22145 ts_tablet_manager.cc:1397] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.003s
I20250112 02:57:30.353241 22145 raft_consensus.cc:357] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.353883 22145 raft_consensus.cc:383] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.353951 21872 tablet_service.cc:1467] Processing CreateTablet for tablet b9034065c6bc435d9f26f85a96297cd3 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.354167 22145 raft_consensus.cc:738] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.354969 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9034065c6bc435d9f26f85a96297cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.364434 22008 tablet_service.cc:1467] Processing CreateTablet for tablet b9034065c6bc435d9f26f85a96297cd3 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.348778 22143 raft_consensus.cc:357] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.365433 22143 raft_consensus.cc:383] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.365746 22143 raft_consensus.cc:738] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.366597 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 62253be16a1a4b2cb7d50264e82478fc (DEFAULT_TABLE table=default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.367516 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62253be16a1a4b2cb7d50264e82478fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.366402 22143 consensus_queue.cc:260] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.377357 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.377403 22145 consensus_queue.cc:260] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:30.378479 21691 leader_election.cc:343] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:30.379346 21691 leader_election.cc:343] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.382308 21899 raft_consensus.cc:3054] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.389010 22143 ts_tablet_manager.cc:1428] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250112 02:57:30.391151 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5 (DEFAULT_TABLE table=default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.392127 22143 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:30.392359 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 2 got 0
I20250112 02:57:30.394177 22902 raft_consensus.cc:491] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.394850 22019 tablet_service.cc:1467] Processing CreateTablet for tablet 62253be16a1a4b2cb7d50264e82478fc (DEFAULT_TABLE table=default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.395205 22144 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:30.394701 22902 raft_consensus.cc:513] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.397632 22902 leader_election.cc:290] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.398479 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.399225 22038 raft_consensus.cc:2463] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:30.400813 21690 leader_election.cc:304] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.403185 22145 ts_tablet_manager.cc:1428] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.052s user 0.005s sys 0.000s
I20250112 02:57:30.407872 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9034065c6bc435d9f26f85a96297cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.413683 22902 raft_consensus.cc:2798] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:30.414037 22902 raft_consensus.cc:491] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:30.414335 22902 raft_consensus.cc:3054] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.416396 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62253be16a1a4b2cb7d50264e82478fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.419088 22145 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:30.425009 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]). Expected version 1 got 0
I20250112 02:57:30.426242 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:30.426964 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.426918 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:30.427434 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:30.438524 22902 raft_consensus.cc:513] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.442950 22143 tablet_bootstrap.cc:654] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.446586 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 62253be16a1a4b2cb7d50264e82478fc (DEFAULT_TABLE table=default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.447450 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62253be16a1a4b2cb7d50264e82478fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.457301 21730 tablet_service.cc:1467] Processing CreateTablet for tablet b9034065c6bc435d9f26f85a96297cd3 (DEFAULT_TABLE table=default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:30.459311 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9034065c6bc435d9f26f85a96297cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:30.460382 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:30.460915 22038 raft_consensus.cc:3054] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:30.461623 21978 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.461979 21978 leader_election.cc:304] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.462595 22902 leader_election.cc:290] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.465308 22730 raft_consensus.cc:2743] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:30.474476 22145 tablet_bootstrap.cc:654] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.479756 22143 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:30.480147 22143 ts_tablet_manager.cc:1397] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.000s
I20250112 02:57:30.483099 22144 tablet_bootstrap.cc:654] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.482728 22143 raft_consensus.cc:357] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.483402 22143 raft_consensus.cc:383] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.483665 22143 raft_consensus.cc:738] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.475595 22038 raft_consensus.cc:2463] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:30.489120 22143 consensus_queue.cc:260] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.492172 22143 ts_tablet_manager.cc:1428] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250112 02:57:30.493016 22143 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:30.505580 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:30.512375 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84f7700b8cce4261a32f2009d1481b99" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:30.513608 21947 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:30.506398 22143 tablet_bootstrap.cc:654] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:30.519660 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 consensus state has no leader
I20250112 02:57:30.520401 22967 sys_catalog.cc:263] Verifying existing consensus state
I20250112 02:57:30.524001 22145 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.524396 22145 ts_tablet_manager.cc:1397] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.003s
I20250112 02:57:30.526356 22145 raft_consensus.cc:357] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.526993 22145 raft_consensus.cc:383] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.527273 22145 raft_consensus.cc:738] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.528600 22143 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:30.532060 22143 ts_tablet_manager.cc:1397] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.002s
I20250112 02:57:30.533068 22145 consensus_queue.cc:260] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.535138 22143 raft_consensus.cc:357] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.535782 22143 raft_consensus.cc:383] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.536039 22143 raft_consensus.cc:738] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.536576 22143 consensus_queue.cc:260] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.538097 21388 catalog_manager.cc:5526] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:30.539006 21690 leader_election.cc:304] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.539698 21388 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:30.541406 21389 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96] alter complete (version 3)
I20250112 02:57:30.542654 21802 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:30.549744 22967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Bootstrap starting.
I20250112 02:57:30.549937 22936 tablet.cc:1721] T 1800d0eb7a9a48d295aca16607291add P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:57:30.553251 21377 catalog_manager.cc:4945] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): alter failed for tablet 36495fb158a440309eddedd3910a7ad3 (table default.Table_142892c5113b4defbea191ff5b1021cf [id=55d68af4c9444c96828bd8d85c3bdaf8]),no further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
I20250112 02:57:30.554231 22143 ts_tablet_manager.cc:1428] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20250112 02:57:30.555065 22143 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:30.557129 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 35) took 4574 ms (client timeout 1000 ms). Trace:
I20250112 02:57:30.560395 22645 raft_consensus.cc:2798] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.560954 21876 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.561303 22919 raft_consensus.cc:491] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.561676 22919 raft_consensus.cc:513] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.563028 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.563498 22919 leader_election.cc:290] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.564041 22090 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:30.564139 22982 tablet.cc:1721] T 1800d0eb7a9a48d295aca16607291add P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:57:30.570122 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 30) took 4573 ms (client timeout 1000 ms). Trace:
W20250112 02:57:30.572955 21691 consensus_peers.cc:487] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:30.573422 21691 leader_election.cc:343] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:30.574081 21691 leader_election.cc:343] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.579133 22645 raft_consensus.cc:695] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.579528 22145 ts_tablet_manager.cc:1428] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.055s user 0.005s sys 0.000s
I20250112 02:57:30.580410 22145 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:30.580456 22645 consensus_queue.cc:237] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.581406 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.581993 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.582260 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.582902 22039 raft_consensus.cc:2463] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:30.583669 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a5c293bbd643aab9409c6e9abe4f0a with delete_type TABLET_DATA_DELETED (replaced by tablet 46f8052651fa496cbb3702c17f592106) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.584461 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.585135 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d699356dd3341338670b433a7fa02ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:22 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.585881 21690 leader_election.cc:304] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.588749 22903 raft_consensus.cc:2798] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:30.589080 22903 raft_consensus.cc:491] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:30.589387 22903 raft_consensus.cc:3054] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.596089 22903 raft_consensus.cc:513] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.598371 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.598407 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.600489 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.601112 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.601542 22145 tablet_bootstrap.cc:654] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.601742 22936 tablet.cc:1721] T cf8bf172abad43ac9a519310334d1640 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:30.604414 21390 rpcz_store.cc:269] 0112 02:57:25.996257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.996352 (+ 95us) service_pool.cc:225] Handling call
0112 02:57:26.021332 (+ 24980us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:30.569951 (+4548619us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":49920256}
I20250112 02:57:30.606876 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f89398c7aa4f2a938a476db897c833 with delete_type TABLET_DATA_DELETED (replaced by tablet e861bab8dcef41328095954a46e00a96) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:30.607161 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 38) took 1116 ms (client timeout 1000 ms). Trace:
W20250112 02:57:30.607503 21390 rpcz_store.cc:269] 0112 02:57:29.490797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.606883 (+1116086us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:30.607100 (+ 217us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250112 02:57:30.597936 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 98aa732d27f144aaad088ecc072837e9 because the tablet was not found. No further retry: Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:30.610688 22012 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.610973 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:30.610993 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 9543a0a0359e4b72bd30f1cb6290f767 because the tablet was not found. No further retry: Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:30.613317 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f787ffe65e34e8f9852d149fff9f947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:30.613647 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 33a5c293bbd643aab9409c6e9abe4f0a because the tablet was not found. No further retry: Not found: Tablet not found: 33a5c293bbd643aab9409c6e9abe4f0a
I20250112 02:57:30.614048 22143 tablet_bootstrap.cc:654] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:30.614305 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 6fbe0e20a157479e9160fc6782e3bf37 because the tablet was not found. No further retry: Not found: Tablet not found: 6fbe0e20a157479e9160fc6782e3bf37
W20250112 02:57:30.614894 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 7d699356dd3341338670b433a7fa02ad because the tablet was not found. No further retry: Not found: Tablet not found: 7d699356dd3341338670b433a7fa02ad
I20250112 02:57:30.614822 22902 raft_consensus.cc:491] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:30.615478 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:30.615341 22902 raft_consensus.cc:513] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:30.616093 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 3c4998c034e54b1bae0ef132cef5b04d because the tablet was not found. No further retry: Not found: Tablet not found: 3c4998c034e54b1bae0ef132cef5b04d
I20250112 02:57:30.616129 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4998c034e54b1bae0ef132cef5b04d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:30.616637 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 36495fb158a440309eddedd3910a7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
W20250112 02:57:30.617254 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5f787ffe65e34e8f9852d149fff9f947 because the tablet was not found. No further retry: Not found: Tablet not found: 5f787ffe65e34e8f9852d149fff9f947
I20250112 02:57:30.617362 22902 leader_election.cc:290] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.617632 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.618320 21809 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:30.618609 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:30.619369 22039 raft_consensus.cc:2463] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:30.620188 21690 leader_election.cc:304] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.620846 22902 raft_consensus.cc:2798] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:30.621007 22769 raft_consensus.cc:491] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.621150 22902 raft_consensus.cc:491] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:30.621433 22902 raft_consensus.cc:3054] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.621419 22769 raft_consensus.cc:513] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.623253 22769 leader_election.cc:290] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:30.623888 21392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 39) took 2828 ms (client timeout 1000 ms). Trace:
I20250112 02:57:30.624701 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.626639 22977 raft_consensus.cc:491] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.626856 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96f98f6840e24f3c9d03937c1bbc00e7" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:30.627295 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 31) took 3619 ms (client timeout 1000 ms). Trace:
I20250112 02:57:30.627689 22039 raft_consensus.cc:2388] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:30.628619 21836 leader_election.cc:304] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:30.629738 22769 raft_consensus.cc:3054] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.630241 22977 raft_consensus.cc:513] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.632928 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.637240 22769 raft_consensus.cc:2743] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:30.639657 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:30.640134 22039 raft_consensus.cc:3054] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:30.646634 22977 leader_election.cc:290] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:30.557482 21389 rpcz_store.cc:269] 0112 02:57:25.982389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:25.982536 (+ 147us) service_pool.cc:225] Handling call
0112 02:57:26.025844 (+ 43308us) spinlock_profiling.cc:243] Waited 40.8 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:30.557052 (+4531208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:27.413536 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:27.413854 (+ 318us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:27.413874 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:27.413886 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:27.414102 (+ 216us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:27.414135 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:27.438191 (+ 24056us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:27.438352 (+ 161us) tablet.cc:698] Row locks acquired
0112 02:57:27.438366 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:27.438473 (+ 107us) write_op.cc:270] Start()
0112 02:57:27.438578 (+ 105us) write_op.cc:276] Timestamp: P: 1736650647438604 usec, L: 0
0112 02:57:27.438594 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:27.439125 (+ 531us) log.cc:844] Serialized 356 byte log entry
0112 02:57:27.747343 (+308218us) op_driver.cc:464] REPLICATION: finished
0112 02:57:27.749347 (+ 2004us) write_op.cc:301] APPLY: starting
0112 02:57:27.749394 (+ 47us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:27.749474 (+ 80us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:27.749559 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:27.749691 (+ 132us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:27.749802 (+ 111us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:27.749839 (+ 37us) write_op.cc:312] APPLY: finished
0112 02:57:27.749985 (+ 146us) log.cc:844] Serialized 23 byte log entry
0112 02:57:27.750100 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:27.750166 (+ 66us) write_op.cc:454] Released schema lock
0112 02:57:27.750193 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":204,"reported_tablets":15,"spinlock_wait_cycles":93784832,"child_traces":[["op",{"apply.queue_time_us":1792,"apply.run_cpu_time_us":1004,"apply.run_wall_time_us":1003,"num_ops":1,"prepare.queue_time_us":13522,"prepare.run_cpu_time_us":1924,"prepare.run_wall_time_us":36212,"raft.queue_time_us":10186,"raft.run_cpu_time_us":21,"raft.run_wall_time_us":23,"replication_time_us":308681}]]}
I20250112 02:57:30.649539 22144 tablet_bootstrap.cc:492] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:30.649490 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.649899 22144 ts_tablet_manager.cc:1397] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.255s user 0.007s sys 0.000s
I20250112 02:57:30.653860 21894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:30.655053 21898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:30.655542 22144 raft_consensus.cc:357] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.656239 22144 raft_consensus.cc:383] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.656489 22144 raft_consensus.cc:738] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.658859 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:30.659852 21980 leader_election.cc:343] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.659804 21895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:30.666056 22903 leader_election.cc:290] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.667522 22039 raft_consensus.cc:2463] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:30.627642 21386 rpcz_store.cc:269] 0112 02:57:27.007577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:27.007700 (+ 123us) service_pool.cc:225] Handling call
0112 02:57:27.071156 (+ 63456us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:30.627148 (+3555992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:28.252102 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:28.252425 (+ 323us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:28.252445 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:28.252456 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:28.256508 (+ 4052us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:28.256546 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:28.256608 (+ 62us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:28.256752 (+ 144us) tablet.cc:698] Row locks acquired
0112 02:57:28.256764 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:28.256867 (+ 103us) write_op.cc:270] Start()
0112 02:57:28.256939 (+ 72us) write_op.cc:276] Timestamp: P: 1736650648256868 usec, L: 0
0112 02:57:28.256952 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:28.257413 (+ 461us) log.cc:844] Serialized 536 byte log entry
0112 02:57:28.268432 (+ 11019us) op_driver.cc:464] REPLICATION: finished
0112 02:57:28.273238 (+ 4806us) write_op.cc:301] APPLY: starting
0112 02:57:28.273290 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:28.273377 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:28.273468 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:28.273614 (+ 146us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:28.273722 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:28.273757 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:57:28.273907 (+ 150us) log.cc:844] Serialized 23 byte log entry
0112 02:57:28.274456 (+ 549us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.274527 (+ 71us) write_op.cc:454] Released schema lock
0112 02:57:28.274556 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:28.274821 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.274845 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":23,"spinlock_wait_cycles":137215232,"child_traces":[["op",{"apply.queue_time_us":4666,"apply.run_cpu_time_us":1092,"apply.run_wall_time_us":1484,"num_ops":1,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":1821,"prepare.run_wall_time_us":6930,"raft.queue_time_us":5619,"raft.run_cpu_time_us":577,"raft.run_wall_time_us":5163,"replication_time_us":11406}]]}
I20250112 02:57:30.669588 21690 leader_election.cc:304] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:30.670913 22903 raft_consensus.cc:2798] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:30.671649 22903 raft_consensus.cc:695] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:30.671864 22931 tablet_replica.cc:331] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:30.672677 22931 raft_consensus.cc:2238] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:30.673113 22931 raft_consensus.cc:2267] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:30.672458 22903 consensus_queue.cc:237] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.676393 22931 ts_tablet_manager.cc:1905] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:30.678442 21691 leader_election.cc:343] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.666669 22144 consensus_queue.cc:260] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:30.679062 21978 leader_election.cc:343] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.679412 21978 leader_election.cc:304] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:30.679759 21691 leader_election.cc:343] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:30.680455 21691 leader_election.cc:343] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.680765 22144 ts_tablet_manager.cc:1428] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20250112 02:57:30.680855 22782 raft_consensus.cc:2743] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:30.675660 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 3c4998c034e54b1bae0ef132cef5b04d because the tablet was not found. No further retry: Not found: Tablet not found: 3c4998c034e54b1bae0ef132cef5b04d
W20250112 02:57:30.690325 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 6fbe0e20a157479e9160fc6782e3bf37 because the tablet was not found. No further retry: Not found: Tablet not found: 6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:30.683161 22902 raft_consensus.cc:513] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.696813 22982 tablet.cc:1721] T cf8bf172abad43ac9a519310334d1640 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:30.700294 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_94ed458009374571ae7a174587b69efc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:30.707243 22989 tablet.cc:1721] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250112 02:57:30.716336 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94ed458009374571ae7a174587b69efc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:30.718690 22144 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:30.721021 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:30.721918 22920 tablet.cc:1721] T 1800d0eb7a9a48d295aca16607291add P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250112 02:57:30.723791 22145 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:30.724169 22145 ts_tablet_manager.cc:1397] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.144s user 0.010s sys 0.010s
I20250112 02:57:30.727190 22145 raft_consensus.cc:357] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.727777 22145 raft_consensus.cc:383] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.728024 22145 raft_consensus.cc:738] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.728616 22145 consensus_queue.cc:260] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.731629 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.733090 22967 log.cc:826] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Log is configured to *not* fsync() on all Append() calls
I20250112 02:57:30.736090 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:30.741124 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 36) took 3488 ms (client timeout 1000 ms). Trace:
W20250112 02:57:30.741459 21383 rpcz_store.cc:269] 0112 02:57:27.252490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:27.318139 (+ 65649us) service_pool.cc:225] Handling call
0112 02:57:27.329773 (+ 11634us) spinlock_profiling.cc:243] Waited 8.64 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:30.740968 (+3411195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":170836,"reported_tablets":17,"spinlock_wait_cycles":19863296}
I20250112 02:57:30.746157 22782 raft_consensus.cc:491] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.746289 22931 ts_tablet_manager.cc:1918] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:30.746623 22931 log.cc:1198] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:30.746543 22782 raft_consensus.cc:513] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.749652 22144 tablet_bootstrap.cc:654] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.751163 22145 ts_tablet_manager.cc:1428] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 02:57:30.751482 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:30.752223 22145 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:30.752641 21980 leader_election.cc:343] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.755237 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.759635 22931 ts_tablet_manager.cc:1939] T c48d1bc147ba48ebbebfd707dc44fe02 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:30.761875 22931 tablet_replica.cc:331] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:30.763258 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet c48d1bc147ba48ebbebfd707dc44fe02 (table default.Table_fffe2a1efd5e4ff3973a842a73b42291 [id=3a57aedad1b843bf9b57633f1ee57d55]) successfully deleted
I20250112 02:57:30.777138 22145 tablet_bootstrap.cc:654] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:30.782390 22144 tablet_bootstrap.cc:492] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:30.783519 21386 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 261) took 2687 ms. Trace:
I20250112 02:57:30.783819 22144 ts_tablet_manager.cc:1397] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.000s
I20250112 02:57:30.783855 21386 rpcz_store.cc:276] 0112 02:57:28.096255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.699870 (+2603615us) service_pool.cc:225] Handling call
0112 02:57:30.756370 (+ 56500us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:30.757665 (+ 1295us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:30.782949 (+ 25284us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:30.783234 (+ 285us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:30.783443 (+ 209us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.764353 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.764739 (+ 386us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.764762 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:30.764779 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:30.765066 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.765108 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.765131 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:30.765327 (+ 196us) tablet.cc:698] Row locks acquired
0112 02:57:30.765344 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:30.765463 (+ 119us) write_op.cc:270] Start()
0112 02:57:30.765543 (+ 80us) write_op.cc:276] Timestamp: P: 1736650650765487 usec, L: 0
0112 02:57:30.765556 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.766035 (+ 479us) log.cc:844] Serialized 433 byte log entry
0112 02:57:30.781273 (+ 15238us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.781523 (+ 250us) write_op.cc:301] APPLY: starting
0112 02:57:30.781568 (+ 45us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.781654 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.781747 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.782155 (+ 408us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.782285 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:30.782320 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:57:30.782462 (+ 142us) log.cc:844] Serialized 29 byte log entry
0112 02:57:30.782625 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.782720 (+ 95us) write_op.cc:454] Released schema lock
0112 02:57:30.782757 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.782961 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.782984 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":1407,"apply.run_wall_time_us":1405,"num_ops":2,"prepare.queue_time_us":5513,"prepare.run_cpu_time_us":1952,"prepare.run_wall_time_us":1954,"raft.queue_time_us":434,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":677,"replication_time_us":15644}]]}
I20250112 02:57:30.786110 22144 raft_consensus.cc:357] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.786911 22144 raft_consensus.cc:383] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:30.787190 22144 raft_consensus.cc:738] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:30.788700 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e86223dc27604b0488bcaf442de062a1" } modify_external_catalogs: true
I20250112 02:57:30.789294 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_e86223dc27604b0488bcaf442de062a1" } modify_external_catalogs: true
I20250112 02:57:30.751159 22782 leader_election.cc:290] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.791976 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:30.800554 21387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 32) took 2694 ms (client timeout 1000 ms). Trace:
W20250112 02:57:30.815639 21384 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 37) took 2505 ms (client timeout 1000 ms). Trace:
I20250112 02:57:30.818717 21383 master_service.cc:946] ReplaceTablet: received request to replace tablet a2ce633c5b3b4cbea17c171564fa72d6 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:30.827348 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf
I20250112 02:57:30.828790 21386 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 263) took 2531 ms. Trace:
I20250112 02:57:30.830834 22144 consensus_queue.cc:260] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.831702 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:30 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:30.829434 21386 rpcz_store.cc:276] 0112 02:57:28.296937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.788404 (+2491467us) service_pool.cc:225] Handling call
0112 02:57:30.789659 (+ 1255us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:30.790270 (+ 611us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:30.790401 (+ 131us) catalog_manager.cc:2762] Locking tablets
0112 02:57:30.790988 (+ 587us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:30.826992 (+ 36004us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:30.827176 (+ 184us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:30.827278 (+ 102us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:30.828737 (+ 1459us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.794091 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.794523 (+ 432us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.794620 (+ 97us) write_op.cc:435] Acquired schema lock
0112 02:57:30.794643 (+ 23us) tablet.cc:659] Decoding operations
0112 02:57:30.794976 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.795023 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.795048 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:30.795349 (+ 301us) tablet.cc:698] Row locks acquired
0112 02:57:30.795365 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:30.795464 (+ 99us) write_op.cc:270] Start()
0112 02:57:30.795541 (+ 77us) write_op.cc:276] Timestamp: P: 1736650650795499 usec, L: 0
0112 02:57:30.795558 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.796026 (+ 468us) log.cc:844] Serialized 734 byte log entry
0112 02:57:30.825309 (+ 29283us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.825534 (+ 225us) write_op.cc:301] APPLY: starting
0112 02:57:30.825597 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.825731 (+ 134us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.825843 (+ 112us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.826147 (+ 304us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.826267 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:30.826305 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:30.826453 (+ 148us) log.cc:844] Serialized 29 byte log entry
0112 02:57:30.826594 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.826651 (+ 57us) write_op.cc:454] Released schema lock
0112 02:57:30.826688 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.826879 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.826906 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":1333,"apply.run_wall_time_us":1333,"num_ops":2,"prepare.queue_time_us":443,"prepare.run_cpu_time_us":2127,"prepare.run_wall_time_us":3069,"raft.queue_time_us":197,"raft.run_cpu_time_us":751,"raft.run_wall_time_us":750,"replication_time_us":29665}]]}
I20250112 02:57:30.832805 22144 ts_tablet_manager.cc:1428] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250112 02:57:30.832906 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.833520 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fbe0e20a157479e9160fc6782e3bf37 with delete_type TABLET_DATA_DELETED (replaced by tablet 698c4b2fd55d43aeb1fd5a3eaf967da5) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.834590 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.836778 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_1761abbaf6d3409ab3979013e59230b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:30.837706 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.838291 22930 tablet_replica.cc:331] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:30.838806 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.839182 22930 raft_consensus.cc:2238] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:30.839602 22930 raft_consensus.cc:2267] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:30.839819 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 33f89398c7aa4f2a938a476db897c833 with delete_type TABLET_DATA_DELETED (replaced by tablet e861bab8dcef41328095954a46e00a96) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.840759 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:30 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:30.841205 22930 ts_tablet_manager.cc:1905] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:30.841317 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 36495fb158a440309eddedd3910a7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
W20250112 02:57:30.841841 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 6fbe0e20a157479e9160fc6782e3bf37 because the tablet was not found. No further retry: Not found: Tablet not found: 6fbe0e20a157479e9160fc6782e3bf37
W20250112 02:57:30.842332 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:30.842922 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 36495fb158a440309eddedd3910a7ad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9c31714d18414800845671d783ac8173) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:30.846382 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 40) took 1922 ms (client timeout 1000 ms). Trace:
W20250112 02:57:30.847054 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1761abbaf6d3409ab3979013e59230b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:30.849519 22931 raft_consensus.cc:2238] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:30.849963 22931 raft_consensus.cc:2267] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:30.851706 22931 ts_tablet_manager.cc:1905] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:30.846701 21391 rpcz_store.cc:269] 0112 02:57:28.923826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:29.612416 (+688590us) service_pool.cc:225] Handling call
0112 02:57:30.846303 (+1233887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.316025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.316539 (+ 514us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.316575 (+ 36us) write_op.cc:435] Acquired schema lock
0112 02:57:30.316603 (+ 28us) tablet.cc:659] Decoding operations
0112 02:57:30.317092 (+ 489us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.317154 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.317175 (+ 21us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:30.317471 (+ 296us) tablet.cc:698] Row locks acquired
0112 02:57:30.317492 (+ 21us) write_op.cc:260] PREPARE: finished
0112 02:57:30.317621 (+ 129us) write_op.cc:270] Start()
0112 02:57:30.317724 (+ 103us) write_op.cc:276] Timestamp: P: 1736650650317871 usec, L: 0
0112 02:57:30.317746 (+ 22us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.318395 (+ 649us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:30.334243 (+ 15848us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.334624 (+ 381us) write_op.cc:301] APPLY: starting
0112 02:57:30.334678 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.334809 (+ 131us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.334959 (+ 150us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.335402 (+ 443us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.335578 (+ 176us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:30.335635 (+ 57us) write_op.cc:312] APPLY: finished
0112 02:57:30.335859 (+ 224us) log.cc:844] Serialized 35 byte log entry
0112 02:57:30.336003 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.336067 (+ 64us) write_op.cc:454] Released schema lock
0112 02:57:30.336110 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.336372 (+ 262us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.336391 (+ 19us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":294375,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":207,"apply.run_cpu_time_us":1713,"apply.run_wall_time_us":1712,"num_ops":3,"prepare.queue_time_us":286,"prepare.run_cpu_time_us":2964,"prepare.run_wall_time_us":3048,"raft.queue_time_us":1432,"raft.run_cpu_time_us":651,"raft.run_wall_time_us":651,"replication_time_us":16398}]]}
W20250112 02:57:30.872795 21385 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 38) took 4274 ms (client timeout 1000 ms). Trace:
I20250112 02:57:30.870162 21391 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ff23deca1f7543abbaab1dcbea953bab" } new_table_name: "default.Table_1ca2eecdba1a42348719978fe8a87766" modify_external_catalogs: true
W20250112 02:57:30.624184 21392 rpcz_store.cc:269] 0112 02:57:27.795605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:27.865203 (+ 69598us) service_pool.cc:225] Handling call
0112 02:57:27.948887 (+ 83684us) spinlock_profiling.cc:243] Waited 78.5 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:30.623704 (+2674817us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:28.470685 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:28.471008 (+ 323us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:28.471029 (+ 21us) write_op.cc:435] Acquired schema lock
0112 02:57:28.471041 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:28.471434 (+ 393us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:28.471472 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:28.471492 (+ 20us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:28.471702 (+ 210us) tablet.cc:698] Row locks acquired
0112 02:57:28.471713 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:28.471874 (+ 161us) write_op.cc:270] Start()
0112 02:57:28.471952 (+ 78us) write_op.cc:276] Timestamp: P: 1736650648471922 usec, L: 0
0112 02:57:28.471966 (+ 14us) op_driver.cc:348] REPLICATION: starting
0112 02:57:28.472404 (+ 438us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:28.488136 (+ 15732us) op_driver.cc:464] REPLICATION: finished
0112 02:57:28.499693 (+ 11557us) write_op.cc:301] APPLY: starting
0112 02:57:28.499751 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:28.499883 (+ 132us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:28.499984 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:28.500308 (+ 324us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:28.500454 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:28.500492 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:28.500697 (+ 205us) log.cc:844] Serialized 35 byte log entry
0112 02:57:28.500838 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.500909 (+ 71us) write_op.cc:454] Released schema lock
0112 02:57:28.500939 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:28.501177 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.501200 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12309,"reported_tablets":27,"spinlock_wait_cycles":180519808,"child_traces":[["op",{"apply.queue_time_us":11398,"apply.run_cpu_time_us":1456,"apply.run_wall_time_us":1456,"num_ops":3,"prepare.queue_time_us":222,"prepare.run_cpu_time_us":1994,"prepare.run_wall_time_us":1994,"raft.queue_time_us":2993,"raft.run_cpu_time_us":516,"raft.run_wall_time_us":516,"replication_time_us":16078}]]}
I20250112 02:57:30.887814 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_eab8f1f56fa94fb6be7fbd91911184bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:30.889477 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eab8f1f56fa94fb6be7fbd91911184bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:30.894523 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 173773us
I20250112 02:57:30.896453 22902 leader_election.cc:290] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:30.898686 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet afeff74abcc643a5ad6565d9dcc40d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:19 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.899367 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.902432 21734 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e81199f79d46d6922eb1c6805d25da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.902628 21725 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.903434 21731 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:30 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:30.902444 21691 leader_election.cc:343] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.908210 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:30.908900 22039 raft_consensus.cc:3054] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:30.912240 21978 leader_election.cc:343] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.912683 21978 leader_election.cc:304] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:30.914342 22730 raft_consensus.cc:2743] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:30.926043 22942 raft_consensus.cc:491] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.926563 22942 raft_consensus.cc:513] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.928421 22942 leader_election.cc:290] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:30.933657 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.943951 21726 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.960615 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2b248224f9416faf4bd83a853ec918" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:30.943920 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:30.962844 21978 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:30.964147 22930 ts_tablet_manager.cc:1918] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:30.964512 22930 log.cc:1198] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:30.965169 22930 ts_tablet_manager.cc:1939] T 96f98f6840e24f3c9d03937c1bbc00e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:30.967268 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet a2ce633c5b3b4cbea17c171564fa72d6
I20250112 02:57:30.967442 22930 tablet_replica.cc:331] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:30.970237 21383 catalog_manager.cc:6596] ReplaceTablet: tablet a2ce633c5b3b4cbea17c171564fa72d6 deleted and replaced by tablet c01590521b6e4ddc9d7fcc7142d49930
I20250112 02:57:30.970688 21383 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50642 (request call id 262) took 2772 ms. Trace:
I20250112 02:57:30.971014 21383 rpcz_store.cc:276] 0112 02:57:28.198624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.818555 (+2619931us) service_pool.cc:225] Handling call
0112 02:57:30.970591 (+152036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.821580 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.821949 (+ 369us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.821973 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:30.821984 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:30.822290 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.822328 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.822351 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:30.822614 (+ 263us) tablet.cc:698] Row locks acquired
0112 02:57:30.822631 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:30.822789 (+ 158us) write_op.cc:270] Start()
0112 02:57:30.822901 (+ 112us) write_op.cc:276] Timestamp: P: 1736650650822822 usec, L: 0
0112 02:57:30.822923 (+ 22us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.823403 (+ 480us) log.cc:844] Serialized 631 byte log entry
0112 02:57:30.857562 (+ 34159us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.857892 (+ 330us) write_op.cc:301] APPLY: starting
0112 02:57:30.857943 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.858044 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.858127 (+ 83us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.858460 (+ 333us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.858684 (+ 224us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:30.858729 (+ 45us) write_op.cc:312] APPLY: finished
0112 02:57:30.858892 (+ 163us) log.cc:844] Serialized 29 byte log entry
0112 02:57:30.859030 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.859100 (+ 70us) write_op.cc:454] Released schema lock
0112 02:57:30.859126 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.859318 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.859341 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":1424,"apply.run_wall_time_us":1422,"num_ops":2,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":2035,"prepare.run_wall_time_us":2036,"replication_time_us":34534}]]}
I20250112 02:57:30.968235 22930 raft_consensus.cc:2238] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:30.980093 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 with delete_type TABLET_DATA_DELETED (replaced by tablet c01590521b6e4ddc9d7fcc7142d49930) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:30.981485 22143 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:30.981907 22143 ts_tablet_manager.cc:1397] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.427s user 0.011s sys 0.000s
I20250112 02:57:30.982569 22930 raft_consensus.cc:2267] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:30.994091 22977 raft_consensus.cc:491] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:30.994539 22977 raft_consensus.cc:513] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:30.998191 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:30.999727 21978 leader_election.cc:343] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.001209 22977 leader_election.cc:290] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:31.006703 22143 raft_consensus.cc:357] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.007361 22143 raft_consensus.cc:383] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.007601 22143 raft_consensus.cc:738] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.008271 22143 consensus_queue.cc:260] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.017135 22645 raft_consensus.cc:491] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.017593 22645 raft_consensus.cc:513] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.019711 22645 leader_election.cc:290] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.024891 22930 ts_tablet_manager.cc:1905] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:31.010413 22143 ts_tablet_manager.cc:1428] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.028s user 0.004s sys 0.002s
I20250112 02:57:30.928368 22039 raft_consensus.cc:2463] T e8f7301bed6c4d98a80c12b70073e62f P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:31.049167 22143 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:31.049217 22987 tablet_replica.cc:331] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:31.059124 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 with delete_type TABLET_DATA_DELETED (replaced by tablet c01590521b6e4ddc9d7fcc7142d49930) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.074270 21690 leader_election.cc:304] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:31.076426 22645 raft_consensus.cc:2798] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:31.076828 22645 raft_consensus.cc:695] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:31.077250 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:31.077795 22045 raft_consensus.cc:2463] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:31.077498 22645 consensus_queue.cc:237] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.082938 22942 raft_consensus.cc:491] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.083315 22942 raft_consensus.cc:513] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.083631 22672 consensus_queue.cc:1035] T f663e183c2ed49b98327b2ad0de12ed9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:31.085389 22942 leader_election.cc:290] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:31.085876 21690 consensus_peers.cc:487] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96f98f6840e24f3c9d03937c1bbc00e7. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:31.087296 21690 leader_election.cc:304] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:31.088299 22672 raft_consensus.cc:2798] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:31.088160 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:31.088644 22672 raft_consensus.cc:491] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:31.089032 22672 raft_consensus.cc:3054] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:31.089851 21978 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.097191 22645 consensus_queue.cc:1035] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:31.098426 22645 consensus_queue.cc:1035] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:31.097122 22672 raft_consensus.cc:513] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.107046 22930 ts_tablet_manager.cc:1918] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:31.108636 22782 raft_consensus.cc:491] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.110224 22782 raft_consensus.cc:513] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.112228 22782 leader_election.cc:290] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:31.112843 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:31.113396 22930 log.cc:1198] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:31.114141 22930 ts_tablet_manager.cc:1939] T 84f7700b8cce4261a32f2009d1481b99 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:31.114193 21978 leader_election.cc:343] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.118430 22931 ts_tablet_manager.cc:1918] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:31.128392 22930 tablet_replica.cc:331] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:31.126060 22931 log.cc:1198] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:31.129518 22930 raft_consensus.cc:2238] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:31.130031 22930 raft_consensus.cc:2267] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:31.131402 22672 leader_election.cc:290] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.130805 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2b248224f9416faf4bd83a853ec918" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.134920 22930 ts_tablet_manager.cc:1905] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:31.135913 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:31.136442 22045 raft_consensus.cc:3054] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:31.137888 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.139202 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.142733 22045 raft_consensus.cc:2463] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:31.150928 21690 leader_election.cc:304] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:31.158730 22930 ts_tablet_manager.cc:1918] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:31.159133 22930 log.cc:1198] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/a2ce633c5b3b4cbea17c171564fa72d6
I20250112 02:57:31.159440 22902 raft_consensus.cc:2798] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:31.159737 22930 ts_tablet_manager.cc:1939] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:31.159885 22902 raft_consensus.cc:695] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:31.160652 22902 consensus_queue.cc:237] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:31.170450 21691 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.174698 22144 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:31.174914 22769 raft_consensus.cc:491] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.175304 22769 raft_consensus.cc:513] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.176887 22769 leader_election.cc:290] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.180828 21894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:31.150197 22931 ts_tablet_manager.cc:1939] T 9543a0a0359e4b72bd30f1cb6290f767 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:31.184001 22931 tablet_replica.cc:331] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:31.184743 22931 raft_consensus.cc:2238] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:31.185154 22931 raft_consensus.cc:2267] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:31.189697 21980 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.190047 21980 leader_election.cc:304] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:31.190946 21980 leader_election.cc:343] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.191282 21980 leader_election.cc:304] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:31.193583 22782 raft_consensus.cc:2743] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:31.194046 22782 raft_consensus.cc:2743] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:31.194234 22145 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:31.196131 21835 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:31.197204 21895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.198271 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.208623 22144 tablet_bootstrap.cc:654] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:31.215813 22145 ts_tablet_manager.cc:1397] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.464s user 0.021s sys 0.036s
I20250112 02:57:31.228425 22931 ts_tablet_manager.cc:1905] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:31.231117 21691 leader_election.cc:343] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:31.233081 21691 consensus_peers.cc:487] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:31.233652 21691 consensus_peers.cc:487] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:31.235430 22920 tablet.cc:1721] T cf8bf172abad43ac9a519310334d1640 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:31.240916 22967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=1 applied=14 ignored=0} inserts{seen=11 ignored=0} mutations{seen=7 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:57:31.242020 22967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Bootstrap complete.
I20250112 02:57:31.245769 22903 raft_consensus.cc:491] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.246218 22903 raft_consensus.cc:513] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.247529 22145 raft_consensus.cc:357] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.248219 22145 raft_consensus.cc:383] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.248241 22903 leader_election.cc:290] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.249959 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:31.250542 22044 raft_consensus.cc:2463] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:31.253448 21690 leader_election.cc:304] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:31.254199 22903 raft_consensus.cc:2798] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:31.254556 22903 raft_consensus.cc:491] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:31.254909 22903 raft_consensus.cc:3054] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:31.260330 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 with delete_type TABLET_DATA_DELETED (replaced by tablet c01590521b6e4ddc9d7fcc7142d49930) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:31.262488 22931 ts_tablet_manager.cc:1918] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:31.248477 22145 raft_consensus.cc:738] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.262869 22931 log.cc:1198] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/98aa732d27f144aaad088ecc072837e9
I20250112 02:57:31.262993 22144 tablet_bootstrap.cc:492] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:31.263427 22144 ts_tablet_manager.cc:1397] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.089s user 0.010s sys 0.000s
I20250112 02:57:31.263341 22145 consensus_queue.cc:260] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.266129 22144 raft_consensus.cc:357] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.280225 22145 ts_tablet_manager.cc:1428] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.064s user 0.005s sys 0.001s
I20250112 02:57:30.992980 21386 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 264) took 2695 ms. Trace:
W20250112 02:57:30.994984 21391 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 750246f0a0c549229d4e7a87cd6ddfd9 consensus state has no leader
W20250112 02:57:30.800920 21387 rpcz_store.cc:269] 0112 02:57:28.105428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:28.124773 (+ 19345us) service_pool.cc:225] Handling call
0112 02:57:30.800202 (+2675429us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:29.629378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:29.629813 (+ 435us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:29.629844 (+ 31us) write_op.cc:435] Acquired schema lock
0112 02:57:29.629857 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:29.630133 (+ 276us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:29.630177 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:29.630200 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:29.630354 (+ 154us) tablet.cc:698] Row locks acquired
0112 02:57:29.630370 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:29.630482 (+ 112us) write_op.cc:270] Start()
0112 02:57:29.636198 (+ 5716us) write_op.cc:276] Timestamp: P: 1736650649630460 usec, L: 0
0112 02:57:29.636224 (+ 26us) op_driver.cc:348] REPLICATION: starting
0112 02:57:29.636965 (+ 741us) log.cc:844] Serialized 536 byte log entry
0112 02:57:29.645478 (+ 8513us) op_driver.cc:464] REPLICATION: finished
0112 02:57:29.645771 (+ 293us) write_op.cc:301] APPLY: starting
0112 02:57:29.645840 (+ 69us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:29.645948 (+ 108us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:29.646088 (+ 140us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:29.646283 (+ 195us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:29.646460 (+ 177us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:29.646542 (+ 82us) write_op.cc:312] APPLY: finished
0112 02:57:29.646748 (+ 206us) log.cc:844] Serialized 23 byte log entry
0112 02:57:29.646877 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:29.646944 (+ 67us) write_op.cc:454] Released schema lock
0112 02:57:29.646974 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:29.647181 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:29.647202 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":160818,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":129,"apply.run_cpu_time_us":1384,"apply.run_wall_time_us":1384,"num_ops":1,"prepare.queue_time_us":663,"prepare.run_cpu_time_us":2456,"prepare.run_wall_time_us":8005,"raft.queue_time_us":3052,"raft.run_cpu_time_us":705,"raft.run_wall_time_us":1109,"replication_time_us":9158,"spinlock_wait_cycles":1201408}]]}
I20250112 02:57:31.288405 22903 raft_consensus.cc:513] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.286732 21386 rpcz_store.cc:276] 0112 02:57:28.297208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.835184 (+2537976us) service_pool.cc:225] Handling call
0112 02:57:30.848640 (+ 13456us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:30.866187 (+ 17547us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:30.992419 (+126232us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:30.992710 (+ 291us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:30.992925 (+ 215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.871758 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.872161 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.872185 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:30.872196 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:30.872514 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.872552 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.872575 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:30.872811 (+ 236us) tablet.cc:698] Row locks acquired
0112 02:57:30.872825 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:30.872946 (+ 121us) write_op.cc:270] Start()
0112 02:57:30.873025 (+ 79us) write_op.cc:276] Timestamp: P: 1736650650873012 usec, L: 0
0112 02:57:30.873044 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.873537 (+ 493us) log.cc:844] Serialized 433 byte log entry
0112 02:57:30.880986 (+ 7449us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.881275 (+ 289us) write_op.cc:301] APPLY: starting
0112 02:57:30.881339 (+ 64us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.881428 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.881517 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.881884 (+ 367us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.882032 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:30.882074 (+ 42us) write_op.cc:312] APPLY: finished
0112 02:57:30.882228 (+ 154us) log.cc:844] Serialized 29 byte log entry
0112 02:57:30.882364 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.882427 (+ 63us) write_op.cc:454] Released schema lock
0112 02:57:30.882456 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.992488 (+110032us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.992514 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"apply.run_cpu_time_us":1419,"apply.run_wall_time_us":111165,"num_ops":2,"prepare.queue_time_us":4465,"prepare.run_cpu_time_us":2055,"prepare.run_wall_time_us":2054,"raft.queue_time_us":74,"raft.run_cpu_time_us":488,"raft.run_wall_time_us":488,"replication_time_us":7845,"spinlock_wait_cycles":147200}]]}
I20250112 02:57:31.289916 21386 master_service.cc:946] ReplaceTablet: received request to replace tablet 7de9ed3d1552490da7ab76037d7debc0 from {username='slave'} at 127.0.0.1:50642
W20250112 02:57:30.816147 21384 rpcz_store.cc:269] 0112 02:57:28.310082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:28.384295 (+ 74213us) service_pool.cc:225] Handling call
0112 02:57:30.815435 (+2431140us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:29.843242 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:29.843635 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:29.843663 (+ 28us) write_op.cc:435] Acquired schema lock
0112 02:57:29.843675 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:29.843947 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:29.843990 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:29.844013 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:29.844160 (+ 147us) tablet.cc:698] Row locks acquired
0112 02:57:29.844171 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:29.844299 (+ 128us) write_op.cc:270] Start()
0112 02:57:29.844385 (+ 86us) write_op.cc:276] Timestamp: P: 1736650649844367 usec, L: 0
0112 02:57:29.844398 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:29.844887 (+ 489us) log.cc:844] Serialized 536 byte log entry
0112 02:57:29.852216 (+ 7329us) op_driver.cc:464] REPLICATION: finished
0112 02:57:29.852419 (+ 203us) write_op.cc:301] APPLY: starting
0112 02:57:29.852470 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:29.852553 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:29.852646 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:29.852784 (+ 138us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:29.852898 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:29.852936 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:29.853109 (+ 173us) log.cc:844] Serialized 23 byte log entry
0112 02:57:29.853227 (+ 118us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:29.853280 (+ 53us) write_op.cc:454] Released schema lock
0112 02:57:29.853306 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:29.853501 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:29.853523 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":161508,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":1033,"apply.run_wall_time_us":1033,"num_ops":1,"prepare.queue_time_us":14578,"prepare.run_cpu_time_us":1903,"prepare.run_wall_time_us":1902,"raft.queue_time_us":1111,"raft.run_cpu_time_us":611,"raft.run_wall_time_us":620,"replication_time_us":7730}]]}
W20250112 02:57:31.046782 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
W20250112 02:57:31.290948 21376 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 244301us
I20250112 02:57:31.291586 22145 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:31.292444 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 9543a0a0359e4b72bd30f1cb6290f767 because the tablet was not found. No further retry: Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767
W20250112 02:57:31.293009 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 33f89398c7aa4f2a938a476db897c833 because the tablet was not found. No further retry: Not found: Tablet not found: 33f89398c7aa4f2a938a476db897c833
I20250112 02:57:31.292728 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2c8b88dcd37b4ecfa44453ce970e61cc" } new_table_name: "default.Table_8f4aa29e292a4231bda19d00c4db1718" modify_external_catalogs: true
W20250112 02:57:31.293529 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 6fbe0e20a157479e9160fc6782e3bf37 because the tablet was not found. No further retry: Not found: Tablet not found: 6fbe0e20a157479e9160fc6782e3bf37
W20250112 02:57:31.294039 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 3c4998c034e54b1bae0ef132cef5b04d because the tablet was not found. No further retry: Not found: Tablet not found: 3c4998c034e54b1bae0ef132cef5b04d
W20250112 02:57:31.295116 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 36495fb158a440309eddedd3910a7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
W20250112 02:57:31.295789 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet afeff74abcc643a5ad6565d9dcc40d62 because the tablet was not found. No further retry: Not found: Tablet not found: afeff74abcc643a5ad6565d9dcc40d62
W20250112 02:57:31.297344 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 99e81199f79d46d6922eb1c6805d25da because the tablet was not found. No further retry: Not found: Tablet not found: 99e81199f79d46d6922eb1c6805d25da
I20250112 02:57:31.297907 22902 raft_consensus.cc:491] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.298363 22902 raft_consensus.cc:513] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.300185 22902 leader_election.cc:290] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:30.873095 21385 rpcz_store.cc:269] 0112 02:57:26.598291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:26.598412 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:26.646711 (+ 48299us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:28.087741 (+1441030us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.087763 (+ 22us) write_op.cc:454] Released schema lock
0112 02:57:30.872622 (+2784859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:28.025327 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:28.025683 (+ 356us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:28.025705 (+ 22us) write_op.cc:435] Acquired schema lock
0112 02:57:28.025719 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:28.025948 (+ 229us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:28.025990 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:28.026023 (+ 33us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:28.026160 (+ 137us) tablet.cc:698] Row locks acquired
0112 02:57:28.026173 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:28.026270 (+ 97us) write_op.cc:270] Start()
0112 02:57:28.026353 (+ 83us) write_op.cc:276] Timestamp: P: 1736650648026338 usec, L: 0
0112 02:57:28.026369 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:28.026827 (+ 458us) log.cc:844] Serialized 536 byte log entry
0112 02:57:28.037428 (+ 10601us) op_driver.cc:464] REPLICATION: finished
0112 02:57:28.037730 (+ 302us) write_op.cc:301] APPLY: starting
0112 02:57:28.037779 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:28.037859 (+ 80us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:28.037956 (+ 97us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:28.038106 (+ 150us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:28.038226 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:28.038263 (+ 37us) write_op.cc:312] APPLY: finished
0112 02:57:28.038412 (+ 149us) log.cc:844] Serialized 23 byte log entry
0112 02:57:28.039158 (+ 746us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:28.039219 (+ 61us) write_op.cc:454] Released schema lock
0112 02:57:28.039246 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":163799,"reported_tablets":28,"spinlock_wait_cycles":95160064,"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1054,"apply.run_wall_time_us":1662,"num_ops":1,"prepare.queue_time_us":367,"prepare.run_cpu_time_us":1779,"prepare.run_wall_time_us":2772,"raft.queue_time_us":310,"raft.run_cpu_time_us":615,"raft.run_wall_time_us":8900,"replication_time_us":10985}]]}
I20250112 02:57:31.259016 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 9543a0a0359e4b72bd30f1cb6290f767 (table default.Table_68a1e71fb38145f797bf5a1740443cb4 [id=76c0e85a35cf42f990746711bb66b706]) successfully deleted
I20250112 02:57:31.301147 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:31.301795 22045 raft_consensus.cc:2463] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:31.301905 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
W20250112 02:57:31.302410 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 6fbe0e20a157479e9160fc6782e3bf37 because the tablet was not found. No further retry: Not found: Tablet not found: 6fbe0e20a157479e9160fc6782e3bf37
I20250112 02:57:31.287981 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_8a6b1335ab9d4d8697519f270c8ac7ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:31.302809 21690 leader_election.cc:304] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:31.303474 22902 raft_consensus.cc:2798] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:31.303788 22902 raft_consensus.cc:491] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:31.304093 22902 raft_consensus.cc:3054] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:31.304191 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a6b1335ab9d4d8697519f270c8ac7ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:31.306207 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_c562ca46d2f349e58930eef499aef1cf" } modify_external_catalogs: true
I20250112 02:57:31.312363 22902 raft_consensus.cc:513] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.315085 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:31.315688 22045 raft_consensus.cc:3054] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:31.316303 22987 raft_consensus.cc:2238] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:31.316814 22987 raft_consensus.cc:2267] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:31.318975 22987 ts_tablet_manager.cc:1905] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:31.325825 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:31.326370 22044 raft_consensus.cc:3054] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:31.332720 22044 raft_consensus.cc:2463] T 2722e4001ba44a7998f4ba7682bd3c44 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:31.348049 22145 tablet_bootstrap.cc:654] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:31.350874 22045 raft_consensus.cc:2463] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:31.360087 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 169236us
W20250112 02:57:31.362252 21980 leader_election.cc:343] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.362758 21980 leader_election.cc:304] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:31.363498 22730 raft_consensus.cc:2743] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:31.364513 21980 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.269588 22931 ts_tablet_manager.cc:1939] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:31.383620 22144 raft_consensus.cc:383] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.334698 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.338568 21895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:31.339872 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:31.385716 22144 raft_consensus.cc:738] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.386462 22144 consensus_queue.cc:260] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.341595 22996 raft_consensus.cc:491] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.271586 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:31.387935 22996 raft_consensus.cc:513] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.391688 22996 leader_election.cc:290] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:31.341074 21836 leader_election.cc:343] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:31.392270 21836 leader_election.cc:304] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
W20250112 02:57:31.392897 21836 raft_consensus.cc:2681] T 98aa732d27f144aaad088ecc072837e9 P cca160b7cffb4ae287ea077e7352ffc9: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 02:57:31.393379 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 52367.1us
I20250112 02:57:31.397627 21902 raft_consensus.cc:3054] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:31.329980 22672 consensus_queue.cc:1035] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:31.333781 21690 leader_election.cc:304] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:31.410509 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 76843.1us
I20250112 02:57:31.411685 21690 leader_election.cc:304] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:31.412343 22919 raft_consensus.cc:2798] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:31.342789 22903 leader_election.cc:290] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.344128 22902 leader_election.cc:290] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.413012 22902 raft_consensus.cc:2798] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:31.413379 22902 raft_consensus.cc:695] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:31.414101 22902 consensus_queue.cc:237] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.054360 22143 tablet_bootstrap.cc:654] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:31.371291 21691 consensus_peers.cc:487] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:31.457617 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 87929.8us
I20250112 02:57:31.390874 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:31.512152 22144 ts_tablet_manager.cc:1428] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.248s user 0.005s sys 0.000s
W20250112 02:57:31.527338 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 69150.3us
W20250112 02:57:31.531818 21691 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:31.532542 21691 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:31.533207 21691 leader_election.cc:343] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:31.533901 21691 leader_election.cc:343] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:31.548745 23001 raft_consensus.cc:491] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:31.549382 23001 raft_consensus.cc:513] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.552241 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "994e1d114e5a4de5aa1d7651907d718e" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:31.553649 23001 leader_election.cc:290] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:31.560169 22144 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:31.068382 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 96f98f6840e24f3c9d03937c1bbc00e7 (table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]) successfully deleted
I20250112 02:57:31.574934 22144 tablet_bootstrap.cc:654] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:31.574748 21377 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 506583us
I20250112 02:57:31.315058 21391 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 265) took 3017 ms. Trace:
I20250112 02:57:31.315158 21392 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 266) took 2915 ms. Trace:
I20250112 02:57:31.578680 21392 rpcz_store.cc:276] 0112 02:57:28.399923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.887419 (+2487496us) service_pool.cc:225] Handling call
0112 02:57:30.889870 (+ 2451us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:31.075073 (+185203us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:31.092262 (+ 17189us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.092286 (+ 24us) write_op.cc:454] Released schema lock
0112 02:57:31.092457 (+ 171us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:31.314883 (+222426us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:31.315112 (+ 229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:31.078798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:31.079211 (+ 413us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:31.079235 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:31.079247 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:31.079548 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:31.079590 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:31.079614 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:31.079826 (+ 212us) tablet.cc:698] Row locks acquired
0112 02:57:31.079837 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:31.079943 (+ 106us) write_op.cc:270] Start()
0112 02:57:31.080021 (+ 78us) write_op.cc:276] Timestamp: P: 1736650651080094 usec, L: 0
0112 02:57:31.080034 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:31.080515 (+ 481us) log.cc:844] Serialized 433 byte log entry
0112 02:57:31.087097 (+ 6582us) op_driver.cc:464] REPLICATION: finished
0112 02:57:31.087381 (+ 284us) write_op.cc:301] APPLY: starting
0112 02:57:31.087427 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:31.087513 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:31.087612 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:31.088056 (+ 444us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:31.088197 (+ 141us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:31.088241 (+ 44us) write_op.cc:312] APPLY: finished
0112 02:57:31.088415 (+ 174us) log.cc:844] Serialized 29 byte log entry
0112 02:57:31.088542 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.088604 (+ 62us) write_op.cc:454] Released schema lock
0112 02:57:31.088630 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":152,"apply.run_cpu_time_us":1427,"apply.run_wall_time_us":1426,"num_ops":2,"prepare.queue_time_us":2611,"prepare.run_cpu_time_us":2004,"prepare.run_wall_time_us":2003,"raft.queue_time_us":1463,"raft.run_cpu_time_us":605,"raft.run_wall_time_us":604,"replication_time_us":6986}]]}
I20250112 02:57:31.342765 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:31.344038 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_c562ca46d2f349e58930eef499aef1cf" } modify_external_catalogs: true
W20250112 02:57:31.583722 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:31.480121 21387 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 267) took 2897 ms. Trace:
I20250112 02:57:31.480553 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 7de9ed3d1552490da7ab76037d7debc0
I20250112 02:57:31.587620 21386 catalog_manager.cc:6596] ReplaceTablet: tablet 7de9ed3d1552490da7ab76037d7debc0 deleted and replaced by tablet b4aff0623f504bf09011ca6de28a3b82
I20250112 02:57:31.587958 21386 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50642 (request call id 268) took 2843 ms. Trace:
I20250112 02:57:31.589305 22919 raft_consensus.cc:695] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:31.566114 22967 raft_consensus.cc:357] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:31.568699 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 98aa732d27f144aaad088ecc072837e9 (table default.Table_84ec605ca7bd4c48b8fc9e590652654c [id=5f96c5056c16402daab66a868b5721d9]) successfully deleted
I20250112 02:57:31.590713 22967 raft_consensus.cc:738] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9ef739ef9dc460ca8287b1d2c6c2217, State: Initialized, Role: FOLLOWER
W20250112 02:57:31.591451 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf already in progress: creating tablet
W20250112 02:57:31.592065 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 36495fb158a440309eddedd3910a7ad3 because the tablet was not found. No further retry: Not found: Tablet not found: 36495fb158a440309eddedd3910a7ad3
I20250112 02:57:31.591883 22967 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 3.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
W20250112 02:57:31.592599 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet a2ce633c5b3b4cbea17c171564fa72d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ce633c5b3b4cbea17c171564fa72d6 already in progress: creating tablet
I20250112 02:57:31.584841 21387 rpcz_store.cc:276] 0112 02:57:28.582558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.287243 (+2704685us) service_pool.cc:225] Handling call
0112 02:57:31.315804 (+ 28561us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:31.458259 (+142455us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:31.479558 (+ 21299us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:31.479839 (+ 281us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:31.480066 (+ 227us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:31.462389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:31.462818 (+ 429us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:31.462842 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:31.462852 (+ 10us) tablet.cc:659] Decoding operations
0112 02:57:31.463670 (+ 818us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:31.463708 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:31.463729 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:31.463926 (+ 197us) tablet.cc:698] Row locks acquired
0112 02:57:31.463942 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:31.464047 (+ 105us) write_op.cc:270] Start()
0112 02:57:31.464125 (+ 78us) write_op.cc:276] Timestamp: P: 1736650651463948 usec, L: 0
0112 02:57:31.464137 (+ 12us) op_driver.cc:348] REPLICATION: starting
0112 02:57:31.464618 (+ 481us) log.cc:844] Serialized 433 byte log entry
0112 02:57:31.477250 (+ 12632us) op_driver.cc:464] REPLICATION: finished
0112 02:57:31.477508 (+ 258us) write_op.cc:301] APPLY: starting
0112 02:57:31.477550 (+ 42us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:31.477650 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:31.477744 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:31.478153 (+ 409us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:31.478284 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:31.478327 (+ 43us) write_op.cc:312] APPLY: finished
0112 02:57:31.478482 (+ 155us) log.cc:844] Serialized 29 byte log entry
0112 02:57:31.478621 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.478689 (+ 68us) write_op.cc:454] Released schema lock
0112 02:57:31.478720 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:31.478946 (+ 226us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.478968 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":141,"apply.run_cpu_time_us":1409,"apply.run_wall_time_us":1408,"num_ops":2,"prepare.queue_time_us":3021,"prepare.run_cpu_time_us":2410,"prepare.run_wall_time_us":3237,"replication_time_us":13029,"spinlock_wait_cycles":4864}]]}
I20250112 02:57:31.595141 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 with delete_type TABLET_DATA_DELETED (replaced by tablet b4aff0623f504bf09011ca6de28a3b82) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:31.595857 22931 tablet_replica.cc:331] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:31.596642 22931 raft_consensus.cc:2238] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:31.597132 22931 raft_consensus.cc:2267] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:31.599207 22931 ts_tablet_manager.cc:1905] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:31.600020 21835 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:57:31.575541 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 84f7700b8cce4261a32f2009d1481b99 (table default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]) successfully deleted
I20250112 02:57:31.600378 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:31.601198 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 02:57:31.601289 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 33f89398c7aa4f2a938a476db897c833 because the tablet was not found. No further retry: Not found: Tablet not found: 33f89398c7aa4f2a938a476db897c833
W20250112 02:57:31.601821 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf already in progress: creating tablet
I20250112 02:57:31.602303 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet a2ce633c5b3b4cbea17c171564fa72d6 (table default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]) successfully deleted
I20250112 02:57:31.575814 21391 rpcz_store.cc:276] 0112 02:57:28.297457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.869824 (+2572367us) service_pool.cc:225] Handling call
0112 02:57:30.883781 (+ 13957us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:30.884404 (+ 623us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:30.884938 (+ 534us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:30.885172 (+ 234us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:30.994593 (+109421us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:31.315004 (+320411us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.933252 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.933667 (+ 415us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.933696 (+ 29us) write_op.cc:435] Acquired schema lock
0112 02:57:30.933713 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:30.934068 (+ 355us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.934122 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.934163 (+ 41us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:30.934349 (+ 186us) tablet.cc:698] Row locks acquired
0112 02:57:30.934374 (+ 25us) write_op.cc:260] PREPARE: finished
0112 02:57:30.934534 (+ 160us) write_op.cc:270] Start()
0112 02:57:30.934632 (+ 98us) write_op.cc:276] Timestamp: P: 1736650650934596 usec, L: 0
0112 02:57:30.934656 (+ 24us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.935320 (+ 664us) log.cc:844] Serialized 405 byte log entry
0112 02:57:30.991227 (+ 55907us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.992737 (+ 1510us) write_op.cc:301] APPLY: starting
0112 02:57:30.992783 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.992858 (+ 75us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.993370 (+ 512us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.993546 (+ 176us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.993693 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:30.993752 (+ 59us) write_op.cc:312] APPLY: finished
0112 02:57:30.993972 (+ 220us) log.cc:844] Serialized 23 byte log entry
0112 02:57:30.994111 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.994162 (+ 51us) write_op.cc:454] Released schema lock
0112 02:57:30.994192 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1381,"apply.run_cpu_time_us":1210,"apply.run_wall_time_us":1607,"num_ops":1,"prepare.queue_time_us":46853,"prepare.run_cpu_time_us":2402,"prepare.run_wall_time_us":2400,"raft.queue_time_us":11945,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":578,"replication_time_us":56470,"thread_start_us":29617,"threads_started":1}]]}
I20250112 02:57:31.606695 22967 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:57:31.609973 21387 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2573b35f681a4ede8201db20c64d179e" } modify_external_catalogs: true
I20250112 02:57:31.610545 21387 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_2573b35f681a4ede8201db20c64d179e" } modify_external_catalogs: true
I20250112 02:57:31.588243 21386 rpcz_store.cc:276] 0112 02:57:28.744521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.289784 (+2545263us) service_pool.cc:225] Handling call
0112 02:57:31.587898 (+298114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:31.454707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:31.455103 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:31.455131 (+ 28us) write_op.cc:435] Acquired schema lock
0112 02:57:31.455149 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:31.455485 (+ 336us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:31.455531 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:31.455559 (+ 28us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:31.455781 (+ 222us) tablet.cc:698] Row locks acquired
0112 02:57:31.455795 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:31.455912 (+ 117us) write_op.cc:270] Start()
0112 02:57:31.456000 (+ 88us) write_op.cc:276] Timestamp: P: 1736650651455809 usec, L: 0
0112 02:57:31.456019 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:31.456559 (+ 540us) log.cc:844] Serialized 665 byte log entry
0112 02:57:31.474041 (+ 17482us) op_driver.cc:464] REPLICATION: finished
0112 02:57:31.477462 (+ 3421us) write_op.cc:301] APPLY: starting
0112 02:57:31.477509 (+ 47us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:31.477628 (+ 119us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:31.477723 (+ 95us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:31.478071 (+ 348us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:31.478285 (+ 214us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:31.478327 (+ 42us) write_op.cc:312] APPLY: finished
0112 02:57:31.478476 (+ 149us) log.cc:844] Serialized 29 byte log entry
0112 02:57:31.478659 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.478712 (+ 53us) write_op.cc:454] Released schema lock
0112 02:57:31.478738 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:31.478963 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.478985 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3282,"apply.run_cpu_time_us":1439,"apply.run_wall_time_us":1488,"mutex_wait_us":4,"num_ops":2,"prepare.queue_time_us":1458,"prepare.run_cpu_time_us":2140,"prepare.run_wall_time_us":2140,"raft.queue_time_us":71,"raft.run_cpu_time_us":596,"raft.run_wall_time_us":595,"replication_time_us":17934,"spinlock_wait_cycles":189952}]]}
W20250112 02:57:31.614421 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 750246f0a0c549229d4e7a87cd6ddfd9 consensus state has no leader
I20250112 02:57:31.616911 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_d173dea95e5c46a5ac100a31efa536d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:31.618072 21388 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 46f8052651fa496cbb3702c17f592106 consensus state has no leader
W20250112 02:57:31.618291 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d173dea95e5c46a5ac100a31efa536d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:31.620934 21388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 33) took 2473 ms (client timeout 1000 ms). Trace:
I20250112 02:57:31.590226 22919 consensus_queue.cc:237] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.669196 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 with delete_type TABLET_DATA_DELETED (replaced by tablet b4aff0623f504bf09011ca6de28a3b82) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:31.677726 23030 consensus_queue.cc:1035] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:31.680650 22144 tablet_bootstrap.cc:492] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:31.681141 22144 ts_tablet_manager.cc:1397] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.121s user 0.011s sys 0.000s
I20250112 02:57:31.701023 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 2 got 0
I20250112 02:57:31.701958 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
W20250112 02:57:31.621224 21388 rpcz_store.cc:269] 0112 02:57:29.147344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:29.791088 (+643744us) service_pool.cc:225] Handling call
0112 02:57:31.620878 (+1829790us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:30.714738 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:30.726969 (+ 12231us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:30.726995 (+ 26us) write_op.cc:435] Acquired schema lock
0112 02:57:30.727007 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:30.727274 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:30.727315 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:30.727341 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:30.727479 (+ 138us) tablet.cc:698] Row locks acquired
0112 02:57:30.727490 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:30.727603 (+ 113us) write_op.cc:270] Start()
0112 02:57:30.727683 (+ 80us) write_op.cc:276] Timestamp: P: 1736650650727609 usec, L: 0
0112 02:57:30.727695 (+ 12us) op_driver.cc:348] REPLICATION: starting
0112 02:57:30.728343 (+ 648us) log.cc:844] Serialized 536 byte log entry
0112 02:57:30.744151 (+ 15808us) op_driver.cc:464] REPLICATION: finished
0112 02:57:30.744428 (+ 277us) write_op.cc:301] APPLY: starting
0112 02:57:30.744478 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:30.744559 (+ 81us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:30.744645 (+ 86us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:30.744809 (+ 164us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:30.744927 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:30.744963 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:30.745113 (+ 150us) log.cc:844] Serialized 23 byte log entry
0112 02:57:30.745229 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.745286 (+ 57us) write_op.cc:454] Released schema lock
0112 02:57:30.745311 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:30.745511 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:30.745530 (+ 19us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":143,"apply.run_cpu_time_us":1061,"apply.run_wall_time_us":1060,"num_ops":1,"prepare.queue_time_us":4036,"prepare.run_cpu_time_us":2184,"prepare.run_wall_time_us":13960,"raft.queue_time_us":1868,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":2623,"replication_time_us":16353}]]}
I20250112 02:57:31.702720 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
W20250112 02:57:31.731685 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 200173us
W20250112 02:57:31.739434 21691 consensus_peers.cc:487] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:31.747321 22143 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:31.747859 22143 ts_tablet_manager.cc:1397] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.699s user 0.014s sys 0.000s
I20250112 02:57:31.750214 22143 raft_consensus.cc:357] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.751015 22143 raft_consensus.cc:383] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.751327 22143 raft_consensus.cc:738] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.751992 22143 consensus_queue.cc:260] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.754060 22143 ts_tablet_manager.cc:1428] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 02:57:31.754922 22143 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:31.751039 22144 raft_consensus.cc:357] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.766114 22144 raft_consensus.cc:383] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.771764 22144 raft_consensus.cc:738] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.772859 22144 consensus_queue.cc:260] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:31.779717 21691 consensus_peers.cc:487] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:31.793227 22987 ts_tablet_manager.cc:1918] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:31.793067 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_a6fb5e40aeb440eeab4c1e01b0350539"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:31.799468 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ace0f644f3f149feaf59940ff5b1d217"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:31.800418 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6fb5e40aeb440eeab4c1e01b0350539 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:31.801015 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace0f644f3f149feaf59940ff5b1d217 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:31.805513 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]). Expected version 2 got 0
I20250112 02:57:31.814630 22144 ts_tablet_manager.cc:1428] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.095s user 0.007s sys 0.000s
I20250112 02:57:31.819337 22144 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:31.821085 22931 ts_tablet_manager.cc:1918] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:31.829573 21383 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_d498875b28e14a3d815a24d80cb3cc7b" } new_table_name: "default.Table_3ef2592713f140de9a0827e1cb98ebc4" modify_external_catalogs: true
W20250112 02:57:31.837965 22114 log.cc:926] Time spent T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Append to log took a long time: real 0.055s user 0.005s sys 0.005s
I20250112 02:57:31.837781 21980 leader_election.cc:304] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:31.838869 21980 raft_consensus.cc:2681] T a2ce633c5b3b4cbea17c171564fa72d6 P e0406c08e9714e87bdd7fd6b66c9756d: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250112 02:57:31.841635 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 477335us
I20250112 02:57:31.865849 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet edfa495bfefc46b998c2f870d9014900
I20250112 02:57:31.868813 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet edfa495bfefc46b998c2f870d9014900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:31.869693 21387 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet f3da552a973945e284a386304f182777
I20250112 02:57:31.871177 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet f3da552a973945e284a386304f182777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:31.872990 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet f3da552a973945e284a386304f182777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:31.876471 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 41369773a0034c65b25fee871e6a25aa (DEFAULT_TABLE table=default.Table_0d55eaca1ca1484a8b8840ad492e7529 [id=30ad19fdfc1246ebbeaf73fff91f925e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:31.877588 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41369773a0034c65b25fee871e6a25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:31.880120 21385 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 270) took 1784 ms. Trace:
I20250112 02:57:31.895064 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:31.896692 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.897141 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.897084 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 with delete_type TABLET_DATA_DELETED (replaced by tablet b4aff0623f504bf09011ca6de28a3b82) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.396684 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:31.896685 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.896683 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.897631 22012 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:31.908098 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 41369773a0034c65b25fee871e6a25aa (DEFAULT_TABLE table=default.Table_0d55eaca1ca1484a8b8840ad492e7529 [id=30ad19fdfc1246ebbeaf73fff91f925e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:31.909179 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41369773a0034c65b25fee871e6a25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:31.918076 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 1cacc670998c4452bd3c03d99a62ad82 (DEFAULT_TABLE table=default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:31.919160 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cacc670998c4452bd3c03d99a62ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:31.793618 22987 log.cc:1198] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:31.925889 22987 ts_tablet_manager.cc:1939] T 994e1d114e5a4de5aa1d7651907d718e P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:31.926690 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 231) took 1025 ms. Trace:
I20250112 02:57:31.926990 22987 rpcz_store.cc:276] 0112 02:57:30.901576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.902526 (+ 950us) service_pool.cc:225] Handling call
0112 02:57:31.049080 (+146554us) ts_tablet_manager.cc:1145] Deleting tablet 994e1d114e5a4de5aa1d7651907d718e
0112 02:57:31.049113 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:31.625980 (+576867us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:31.793011 (+167031us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:31.926636 (+133625us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":258643,"tablet-delete.queue_time_us":146103}
I20250112 02:57:31.931557 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 1cacc670998c4452bd3c03d99a62ad82 (DEFAULT_TABLE table=default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:31.932579 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cacc670998c4452bd3c03d99a62ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:31.373196 22145 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:31.944145 22145 ts_tablet_manager.cc:1397] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.653s user 0.007s sys 0.003s
I20250112 02:57:31.949805 22145 raft_consensus.cc:357] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.951226 22145 raft_consensus.cc:383] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:31.951597 22145 raft_consensus.cc:738] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:31.952374 22145 consensus_queue.cc:260] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:31.901665 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_40b78fd5c6bf456ba1ffd7b2941251ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:31.958704 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40b78fd5c6bf456ba1ffd7b2941251ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:31.902967 21383 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7d8db14a8054ae69daf42dd76f35511 consensus state has no leader
I20250112 02:57:31.928593 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 994e1d114e5a4de5aa1d7651907d718e (table default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]) successfully deleted
I20250112 02:57:31.929417 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 230) took 1032 ms. Trace:
I20250112 02:57:31.966434 22987 rpcz_store.cc:276] 0112 02:57:30.897387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.897491 (+ 104us) service_pool.cc:225] Handling call
0112 02:57:31.929046 (+1031555us) ts_tablet_manager.cc:1145] Deleting tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf
0112 02:57:31.929090 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:31.929365 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025213}
I20250112 02:57:31.968675 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 228) took 1071 ms. Trace:
I20250112 02:57:31.969122 22987 rpcz_store.cc:276] 0112 02:57:30.896866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.903166 (+ 6300us) service_pool.cc:225] Handling call
0112 02:57:31.968161 (+1064995us) ts_tablet_manager.cc:1145] Deleting tablet 98aa732d27f144aaad088ecc072837e9
0112 02:57:31.968213 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:31.968445 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030623}
I20250112 02:57:31.970762 22987 tablet_replica.cc:331] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:31.972412 21879 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1f4a9244c4b40ad8557de5b36a40e (DEFAULT_TABLE table=default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:31.973358 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce1f4a9244c4b40ad8557de5b36a40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:31.822932 22931 log.cc:1198] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/7de9ed3d1552490da7ab76037d7debc0
I20250112 02:57:31.989399 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet edfa495bfefc46b998c2f870d9014900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:31.880504 21385 rpcz_store.cc:276] 0112 02:57:30.095820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.305825 (+1210005us) service_pool.cc:225] Handling call
0112 02:57:31.582967 (+277142us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:31.584558 (+ 1591us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:31.584680 (+ 122us) catalog_manager.cc:2762] Locking tablets
0112 02:57:31.704157 (+119477us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:31.733889 (+ 29732us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:31.865676 (+131787us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:31.865774 (+ 98us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:31.880052 (+ 14278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:31.710118 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:31.711130 (+ 1012us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:31.711155 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:31.711170 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:31.711987 (+ 817us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:31.712028 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:31.712049 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:31.712348 (+ 299us) tablet.cc:698] Row locks acquired
0112 02:57:31.712365 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:31.712473 (+ 108us) write_op.cc:270] Start()
0112 02:57:31.712557 (+ 84us) write_op.cc:276] Timestamp: P: 1736650651712273 usec, L: 0
0112 02:57:31.712575 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:31.713056 (+ 481us) log.cc:844] Serialized 734 byte log entry
0112 02:57:31.731640 (+ 18584us) op_driver.cc:464] REPLICATION: finished
0112 02:57:31.732503 (+ 863us) write_op.cc:301] APPLY: starting
0112 02:57:31.732552 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:31.732653 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:31.732754 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:31.733004 (+ 250us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:31.733137 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:31.733180 (+ 43us) write_op.cc:312] APPLY: finished
0112 02:57:31.733363 (+ 183us) log.cc:844] Serialized 29 byte log entry
0112 02:57:31.733499 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.733564 (+ 65us) write_op.cc:454] Released schema lock
0112 02:57:31.733593 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:31.740709 (+ 7116us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.740981 (+ 272us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":723,"apply.run_cpu_time_us":1267,"apply.run_wall_time_us":1267,"num_ops":2,"prepare.queue_time_us":4187,"prepare.run_cpu_time_us":2299,"prepare.run_wall_time_us":30004,"raft.queue_time_us":11400,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":628,"replication_time_us":18983}]]}
I20250112 02:57:32.032269 22144 tablet_bootstrap.cc:654] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.035221 22930 tablet_replica.cc:331] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:32.001484 22931 ts_tablet_manager.cc:1939] T 7de9ed3d1552490da7ab76037d7debc0 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:32.046099 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 7de9ed3d1552490da7ab76037d7debc0 (table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]) successfully deleted
W20250112 02:57:32.047214 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet f3da552a973945e284a386304f182777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3da552a973945e284a386304f182777 already in progress: creating tablet
W20250112 02:57:32.003209 21691 consensus_peers.cc:487] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:32.063310 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 60249.3us
I20250112 02:57:32.064808 23021 raft_consensus.cc:491] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.065253 23021 raft_consensus.cc:513] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.067171 23021 leader_election.cc:290] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:32.069025 21836 leader_election.cc:343] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 994e1d114e5a4de5aa1d7651907d718e
W20250112 02:57:32.068991 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 212141us
I20250112 02:57:32.069372 21836 leader_election.cc:304] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.070363 21836 leader_election.cc:304] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.072232 22145 ts_tablet_manager.cc:1428] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.128s user 0.008s sys 0.004s
I20250112 02:57:32.074556 22145 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:32.082845 22977 raft_consensus.cc:491] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.083360 22977 raft_consensus.cc:513] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.085269 22977 leader_election.cc:290] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:32.089083 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:32.089987 21757 raft_consensus.cc:2463] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
W20250112 02:57:32.107456 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e9f8cbfcd4c41e2bdf8c04216a318bf already in progress: creating tablet
I20250112 02:57:31.966023 22143 tablet_bootstrap.cc:654] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.126763 23035 raft_consensus.cc:491] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.126874 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet edfa495bfefc46b998c2f870d9014900 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:32.127218 23035 raft_consensus.cc:513] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.129088 23035 leader_election.cc:290] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:32.130573 23001 raft_consensus.cc:2743] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.136653 21978 leader_election.cc:304] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:32.143428 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet f3da552a973945e284a386304f182777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:32.143994 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 41369773a0034c65b25fee871e6a25aa (DEFAULT_TABLE table=default.Table_0d55eaca1ca1484a8b8840ad492e7529 [id=30ad19fdfc1246ebbeaf73fff91f925e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:32.144536 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 98aa732d27f144aaad088ecc072837e9 because the tablet was not found. No further retry: Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
I20250112 02:57:32.112694 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]). Expected version 1 got 0
I20250112 02:57:32.145262 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41369773a0034c65b25fee871e6a25aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:32.145658 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]). Expected version 2 got 0
I20250112 02:57:32.147212 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 1cacc670998c4452bd3c03d99a62ad82 (DEFAULT_TABLE table=default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:32.147543 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:32.148017 21390 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:32.148303 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cacc670998c4452bd3c03d99a62ad82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:32.127461 21383 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e2b248224f9416faf4bd83a853ec918 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:32.138245 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_142892c5113b4defbea191ff5b1021cf" } modify_external_catalogs: true
I20250112 02:57:32.149936 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_142892c5113b4defbea191ff5b1021cf" } modify_external_catalogs: true
I20250112 02:57:32.142059 21385 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_6e4197a227d046569987a0c17723f994"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:32.151497 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1f4a9244c4b40ad8557de5b36a40e (DEFAULT_TABLE table=default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:32.130581 22996 raft_consensus.cc:2743] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.152781 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce1f4a9244c4b40ad8557de5b36a40e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:32.157068 21385 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e4197a227d046569987a0c17723f994 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:32.153419 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:32.160917 21691 leader_election.cc:343] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.161809 22987 raft_consensus.cc:2238] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
W20250112 02:57:32.168200 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.188:35683, user_credentials={real_user=slave}} blocked reactor thread for 66074.5us
W20250112 02:57:32.178670 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 107425us
W20250112 02:57:32.179996 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet edfa495bfefc46b998c2f870d9014900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfa495bfefc46b998c2f870d9014900 already in progress: creating tablet
I20250112 02:57:32.188525 22977 raft_consensus.cc:2798] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:32.188948 22977 raft_consensus.cc:491] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:32.189329 22977 raft_consensus.cc:3054] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.162629 22987 raft_consensus.cc:2267] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:32.193207 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 49220.8us
W20250112 02:57:32.193972 21980 consensus_peers.cc:487] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:32.194921 22987 ts_tablet_manager.cc:1905] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.198874 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 4ce1f4a9244c4b40ad8557de5b36a40e (DEFAULT_TABLE table=default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:32.199918 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce1f4a9244c4b40ad8557de5b36a40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:32.205196 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 9c31714d18414800845671d783ac8173
I20250112 02:57:32.207996 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c31714d18414800845671d783ac8173 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:32.208259 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 71808.1us
I20250112 02:57:32.209766 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c31714d18414800845671d783ac8173 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:32.209865 21894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.214727 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.219228 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:32.223946 21980 leader_election.cc:343] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:32.225123 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 9c31714d18414800845671d783ac8173 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c31714d18414800845671d783ac8173 already in progress: creating tablet
I20250112 02:57:32.244879 23042 raft_consensus.cc:491] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.245375 23042 raft_consensus.cc:513] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.248997 23042 leader_election.cc:290] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.211550 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:32.216399 21690 consensus_peers.cc:487] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84f7700b8cce4261a32f2009d1481b99. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:32.221818 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c31714d18414800845671d783ac8173 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:32.222896 22143 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:32.252206 22143 ts_tablet_manager.cc:1397] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.497s user 0.008s sys 0.005s
I20250112 02:57:32.252286 23021 consensus_queue.cc:1035] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:32.255097 22143 raft_consensus.cc:357] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.255904 22143 raft_consensus.cc:383] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.256249 22143 raft_consensus.cc:738] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.256968 22143 consensus_queue.cc:260] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:32.258869 21980 leader_election.cc:343] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.238929 21802 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:32.260910 22090 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:32.266153 22143 ts_tablet_manager.cc:1428] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20250112 02:57:32.266448 21978 leader_election.cc:343] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.266803 21978 leader_election.cc:304] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.267048 22143 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:32.267521 23037 raft_consensus.cc:2743] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.269487 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.271318 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e3890251e644c1eb596de043b63491e" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:32.272611 21835 leader_election.cc:343] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: Tablet not RUNNING: STOPPED
W20250112 02:57:32.278272 21691 consensus_peers.cc:487] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:32.288270 22144 tablet_bootstrap.cc:492] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:32.288687 22144 ts_tablet_manager.cc:1397] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.470s user 0.009s sys 0.000s
I20250112 02:57:32.291265 21836 leader_election.cc:304] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.292215 23042 raft_consensus.cc:2743] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.292912 22144 raft_consensus.cc:357] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.293515 22144 raft_consensus.cc:383] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.293741 22144 raft_consensus.cc:738] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.299993 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7d88c7e3065640629e9d17b64174c9e4" } modify_external_catalogs: true
I20250112 02:57:32.300521 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_7d88c7e3065640629e9d17b64174c9e4" } modify_external_catalogs: true
W20250112 02:57:32.302513 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 50774.6us
I20250112 02:57:32.304481 22977 raft_consensus.cc:513] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.306222 22977 leader_election.cc:290] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:32.307731 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:32.308036 22930 raft_consensus.cc:2238] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:32.308264 21757 raft_consensus.cc:3054] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.308539 22930 raft_consensus.cc:2267] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.308969 22045 raft_consensus.cc:2388] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1c2770701ca54288956df9d910d7d3d2 in current term 1: Already voted for candidate e0406c08e9714e87bdd7fd6b66c9756d in this term.
I20250112 02:57:32.311175 22930 ts_tablet_manager.cc:1905] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.314452 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:32.316462 21757 raft_consensus.cc:2463] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
W20250112 02:57:32.316957 21980 leader_election.cc:343] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.323725 23042 raft_consensus.cc:491] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.324146 23042 raft_consensus.cc:513] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.325862 23042 leader_election.cc:290] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.326895 22145 tablet_bootstrap.cc:654] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.327576 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:32.328481 21386 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_311eae8aadc143c09b9dfa448ba3f133"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:32.329936 21386 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_311eae8aadc143c09b9dfa448ba3f133 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:32.332227 21902 raft_consensus.cc:3054] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.332175 21978 leader_election.cc:304] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.338189 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a8273b629804f1c86a813b04747a1aa" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.338860 22044 raft_consensus.cc:2388] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:32.339036 22977 raft_consensus.cc:2798] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:32.339661 22977 raft_consensus.cc:695] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:32.339919 21836 leader_election.cc:304] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.340437 22977 consensus_queue.cc:237] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.341254 23001 raft_consensus.cc:2743] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.294286 22144 consensus_queue.cc:260] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.352077 23030 consensus_queue.cc:1035] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:32.366886 22143 tablet_bootstrap.cc:654] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.384383 21690 leader_election.cc:304] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.388023 22144 ts_tablet_manager.cc:1428] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
I20250112 02:57:32.388944 22144 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:32.391959 23055 raft_consensus.cc:2743] T 118ef6d7c16941598177b92a94c4b472 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.406193 23042 raft_consensus.cc:491] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.406314 22930 ts_tablet_manager.cc:1918] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:32.406841 22930 log.cc:1198] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:32.406654 23042 raft_consensus.cc:513] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.407815 22930 ts_tablet_manager.cc:1939] T f445a4bf43db4571b6d3a4afe8fe823a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:32.409220 22930 tablet_replica.cc:331] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:32.410672 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.411015 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df734ba87264abd803237a5bbb32da2" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:32.411203 22930 raft_consensus.cc:2238] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:32.411772 22930 raft_consensus.cc:2267] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.411923 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet f445a4bf43db4571b6d3a4afe8fe823a (table default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]) successfully deleted
I20250112 02:57:32.412000 23042 leader_election.cc:290] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.412847 21835 leader_election.cc:304] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:32.413678 23042 raft_consensus.cc:2743] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.417438 23055 consensus_queue.cc:1035] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:32.429008 22930 ts_tablet_manager.cc:1905] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.440436 22145 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:32.441125 22145 ts_tablet_manager.cc:1397] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.367s user 0.010s sys 0.002s
I20250112 02:57:32.460167 22145 raft_consensus.cc:357] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.462620 22145 raft_consensus.cc:383] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.462934 22145 raft_consensus.cc:738] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.463104 21388 catalog_manager.cc:5526] T f663e183c2ed49b98327b2ad0de12ed9 P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } }
I20250112 02:57:32.463505 22145 consensus_queue.cc:260] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:32.464465 21691 consensus_peers.cc:487] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:32.465755 22145 ts_tablet_manager.cc:1428] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.024s user 0.001s sys 0.004s
I20250112 02:57:32.466719 22145 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:32.474468 22930 ts_tablet_manager.cc:1918] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:32.477053 22930 log.cc:1198] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/5e3890251e644c1eb596de043b63491e
I20250112 02:57:32.477828 22930 ts_tablet_manager.cc:1939] T 5e3890251e644c1eb596de043b63491e P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:32.484505 22987 ts_tablet_manager.cc:1918] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:32.484870 22987 log.cc:1198] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/5e3890251e644c1eb596de043b63491e
I20250112 02:57:32.485530 22987 ts_tablet_manager.cc:1939] T 5e3890251e644c1eb596de043b63491e P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
W20250112 02:57:32.488291 21690 consensus_peers.cc:487] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f445a4bf43db4571b6d3a4afe8fe823a. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:32.514699 22930 tablet_replica.cc:331] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:32.515590 22930 raft_consensus.cc:2238] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:32.516494 22930 raft_consensus.cc:2267] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.519117 22930 ts_tablet_manager.cc:1905] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.528455 22144 tablet_bootstrap.cc:654] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.477572 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 922b22f0b1284bc4a732371a61cb1ed4
I20250112 02:57:32.539813 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:32.539853 21388 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:32.540925 21388 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:32.543991 21902 raft_consensus.cc:3054] T 0df734ba87264abd803237a5bbb32da2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.522611 22145 tablet_bootstrap.cc:654] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:32.546605 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 57826.4us
I20250112 02:57:32.547708 23030 consensus_queue.cc:1035] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:32.549748 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:32.507807 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 5e3890251e644c1eb596de043b63491e (table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]) successfully deleted
W20250112 02:57:32.555714 21377 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 47954.8us
I20250112 02:57:32.542052 21385 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_3c0edbf3cfc84aa29bbe9a7789e65aec" } new_table_name: "default.Table_e32d5645fc44476a881643b011a69542" modify_external_catalogs: true
I20250112 02:57:32.559063 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:32.478125 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 4e2b248224f9416faf4bd83a853ec918
W20250112 02:57:32.549922 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 922b22f0b1284bc4a732371a61cb1ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 922b22f0b1284bc4a732371a61cb1ed4 already in progress: creating tablet
I20250112 02:57:32.562419 21383 catalog_manager.cc:6596] ReplaceTablet: tablet 4e2b248224f9416faf4bd83a853ec918 deleted and replaced by tablet d1e0548276474b68919cb10fbcfcdf04
I20250112 02:57:32.562376 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:32.563445 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:32.564875 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 4e2b248224f9416faf4bd83a853ec918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e2b248224f9416faf4bd83a853ec918 already in progress: creating tablet
I20250112 02:57:32.565091 21383 master_service.cc:946] ReplaceTablet: received request to replace tablet 15c16031f5b74cf9885fac110f445e1a from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:32.562574 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:32.486249 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 232) took 1584 ms. Trace:
I20250112 02:57:32.578677 22987 rpcz_store.cc:276] 0112 02:57:30.901877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.943845 (+ 41968us) service_pool.cc:225] Handling call
0112 02:57:31.970580 (+1026735us) ts_tablet_manager.cc:1145] Deleting tablet 5e3890251e644c1eb596de043b63491e
0112 02:57:31.970629 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:32.239155 (+268526us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:32.483724 (+244569us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:32.486198 (+ 2474us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32350,"tablet-delete.queue_time_us":1024138}
I20250112 02:57:32.579995 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 5e3890251e644c1eb596de043b63491e (table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]) successfully deleted
I20250112 02:57:32.590030 22144 tablet_bootstrap.cc:492] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:32.590914 22145 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:32.591327 22145 ts_tablet_manager.cc:1397] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.011s
I20250112 02:57:32.594374 22145 raft_consensus.cc:357] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.595057 22145 raft_consensus.cc:383] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.595299 22145 raft_consensus.cc:738] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.598703 22145 consensus_queue.cc:260] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.590452 22144 ts_tablet_manager.cc:1397] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.202s user 0.004s sys 0.004s
I20250112 02:57:32.609108 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 229) took 1712 ms. Trace:
I20250112 02:57:32.609488 22987 rpcz_store.cc:276] 0112 02:57:30.897110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.899178 (+ 2068us) service_pool.cc:225] Handling call
0112 02:57:32.608844 (+1709666us) ts_tablet_manager.cc:1145] Deleting tablet 0bca1c68aae24a1f9974a0bd2244fbf8
0112 02:57:32.608889 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:32.609054 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647169}
W20250112 02:57:32.610760 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 0bca1c68aae24a1f9974a0bd2244fbf8 because the tablet was not found. No further retry: Not found: Tablet not found: 0bca1c68aae24a1f9974a0bd2244fbf8
I20250112 02:57:32.611935 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 234) took 1642 ms. Trace:
I20250112 02:57:32.612195 22143 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:32.612577 22143 ts_tablet_manager.cc:1397] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.346s user 0.011s sys 0.002s
I20250112 02:57:32.612298 22987 rpcz_store.cc:276] 0112 02:57:30.969068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.975160 (+ 6092us) service_pool.cc:225] Handling call
0112 02:57:32.611545 (+1636385us) ts_tablet_manager.cc:1145] Deleting tablet a2ce633c5b3b4cbea17c171564fa72d6
0112 02:57:32.611583 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:32.611887 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631068}
W20250112 02:57:32.613608 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet a2ce633c5b3b4cbea17c171564fa72d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ce633c5b3b4cbea17c171564fa72d6 already in progress: creating tablet
I20250112 02:57:32.622354 22143 raft_consensus.cc:357] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.623064 22143 raft_consensus.cc:383] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.623309 22143 raft_consensus.cc:738] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.623847 22143 consensus_queue.cc:260] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.625834 22143 ts_tablet_manager.cc:1428] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250112 02:57:32.627568 22144 raft_consensus.cc:357] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.627403 21391 catalog_manager.cc:5526] T 2cbf5e2a26474c98b9cb48b5bbdceafb P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.628239 22144 raft_consensus.cc:383] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.628512 22144 raft_consensus.cc:738] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.629222 22144 consensus_queue.cc:260] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.638449 23030 consensus_queue.cc:1035] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:32.652150 23022 raft_consensus.cc:491] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.600687 22145 ts_tablet_manager.cc:1428] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250112 02:57:32.652554 23022 raft_consensus.cc:513] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.653445 22145 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:32.655200 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:32.656337 21391 catalog_manager.cc:5526] T 0df734ba87264abd803237a5bbb32da2 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.657764 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:32.658758 21391 catalog_manager.cc:5526] T 67260685978247898ad224128ed8964f P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.661216 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.661842 23022 leader_election.cc:290] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:32.662272 21691 leader_election.cc:343] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.663931 21391 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:32.671180 22144 ts_tablet_manager.cc:1428] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.068s user 0.005s sys 0.001s
I20250112 02:57:32.671757 22143 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:32.689267 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.689960 22045 raft_consensus.cc:2463] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:32.691254 21690 leader_election.cc:304] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.692366 23030 raft_consensus.cc:2798] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:32.692750 23030 raft_consensus.cc:491] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:32.693142 23030 raft_consensus.cc:3054] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.708827 22987 tablet_replica.cc:331] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:32.712749 21387 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:32.715263 22987 raft_consensus.cc:2238] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:32.715852 22987 raft_consensus.cc:2267] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.716830 22144 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:32.721953 22144 tablet_bootstrap.cc:654] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.730758 22145 tablet_bootstrap.cc:654] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.732447 22143 tablet_bootstrap.cc:654] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.740010 22977 raft_consensus.cc:491] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.740454 22977 raft_consensus.cc:513] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.741998 22930 ts_tablet_manager.cc:1918] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:32.742254 22977 leader_election.cc:290] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:32.745265 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:32.742372 22930 log.cc:1198] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/7de9ed3d1552490da7ab76037d7debc0
W20250112 02:57:32.746376 21978 leader_election.cc:343] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.745893 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.747107 22930 ts_tablet_manager.cc:1939] T 7de9ed3d1552490da7ab76037d7debc0 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:32.747711 23070 raft_consensus.cc:491] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.748131 23070 raft_consensus.cc:513] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:32.748013 21980 leader_election.cc:343] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.748808 21387 catalog_manager.cc:5526] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.749390 21980 leader_election.cc:304] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.750089 21387 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:32.750774 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 7de9ed3d1552490da7ab76037d7debc0 (table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]) successfully deleted
I20250112 02:57:32.749902 23070 leader_election.cc:290] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.751662 21387 catalog_manager.cc:5526] T 2722e4001ba44a7998f4ba7682bd3c44 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.752210 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.752676 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.753250 22039 raft_consensus.cc:2463] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:32.753427 21691 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.753959 22977 raft_consensus.cc:2743] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.753647 21387 catalog_manager.cc:5526] T e8f7301bed6c4d98a80c12b70073e62f P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.754195 21690 leader_election.cc:304] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.755708 23022 raft_consensus.cc:2798] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:32.756047 23022 raft_consensus.cc:491] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:32.756361 23022 raft_consensus.cc:3054] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.768916 22143 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:32.769755 22143 ts_tablet_manager.cc:1397] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.143s user 0.009s sys 0.000s
I20250112 02:57:32.779608 23022 raft_consensus.cc:513] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.781237 22143 raft_consensus.cc:357] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.781821 22143 raft_consensus.cc:383] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.782040 22143 raft_consensus.cc:738] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.782073 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:32.782567 22143 consensus_queue.cc:260] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:32.783367 21691 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.784538 22145 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:32.784639 22977 raft_consensus.cc:491] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.784910 22145 ts_tablet_manager.cc:1397] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.132s user 0.009s sys 0.007s
I20250112 02:57:32.785071 22977 raft_consensus.cc:513] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.787036 22977 leader_election.cc:290] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:32.787882 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.788705 22144 tablet_bootstrap.cc:492] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:32.784230 23022 leader_election.cc:290] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:32.788928 21980 leader_election.cc:343] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.789088 22144 ts_tablet_manager.cc:1397] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.003s
I20250112 02:57:32.789597 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:32.790724 21978 leader_election.cc:343] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.791124 22930 tablet_replica.cc:331] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:32.791086 21978 leader_election.cc:304] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.791270 22144 raft_consensus.cc:357] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.791859 22144 raft_consensus.cc:383] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.791991 22977 raft_consensus.cc:2743] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:32.792104 22144 raft_consensus.cc:738] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.792640 22144 consensus_queue.cc:260] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.791886 22145 raft_consensus.cc:357] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:32.754982 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because the tablet was not found. No further retry: Not found: Tablet not found: 84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:32.794405 22145 raft_consensus.cc:383] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.794623 22145 raft_consensus.cc:738] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:32.795202 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because the tablet was not found. No further retry: Not found: Tablet not found: 96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:32.795291 22930 raft_consensus.cc:2238] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:32.795724 22930 raft_consensus.cc:2267] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:32.795773 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
W20250112 02:57:32.796249 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet edfa495bfefc46b998c2f870d9014900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfa495bfefc46b998c2f870d9014900 already in progress: creating tablet
W20250112 02:57:32.796770 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet f3da552a973945e284a386304f182777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3da552a973945e284a386304f182777 already in progress: creating tablet
W20250112 02:57:32.797272 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 9c31714d18414800845671d783ac8173 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c31714d18414800845671d783ac8173 already in progress: creating tablet
W20250112 02:57:32.797762 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 922b22f0b1284bc4a732371a61cb1ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 922b22f0b1284bc4a732371a61cb1ed4 already in progress: creating tablet
W20250112 02:57:32.787890 21376 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e7d8db14a8054ae69daf42dd76f35511 (table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e7d8db14a8054ae69daf42dd76f35511 consensus state has no leader
I20250112 02:57:32.799505 21387 catalog_manager.cc:5526] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:32.800714 21387 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:32.801463 22145 consensus_queue.cc:260] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.804366 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2601daad8ba742ec88ccfe603440e8e7" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:32.804862 22039 raft_consensus.cc:3054] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.811208 22039 raft_consensus.cc:2463] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:32.811697 22145 ts_tablet_manager.cc:1428] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250112 02:57:32.812505 22145 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:32.812301 21690 leader_election.cc:304] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.815179 22977 raft_consensus.cc:491] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.815610 22977 raft_consensus.cc:513] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.817000 22143 ts_tablet_manager.cc:1428] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.047s user 0.004s sys 0.002s
I20250112 02:57:32.817863 22143 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:32.818895 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:32.820034 21980 leader_election.cc:343] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.794471 22144 ts_tablet_manager.cc:1428] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:57:32.824813 22144 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:32.824957 22930 ts_tablet_manager.cc:1905] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.827133 22977 leader_election.cc:290] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:32.833177 23022 raft_consensus.cc:2798] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:32.834033 23022 raft_consensus.cc:695] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:32.835541 23022 consensus_queue.cc:237] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.858814 22145 tablet_bootstrap.cc:654] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.863351 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:32.864706 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:32.866709 21383 catalog_manager.cc:6596] ReplaceTablet: tablet 15c16031f5b74cf9885fac110f445e1a deleted and replaced by tablet da54d9dbab2f416785ec0379095bcf3e
I20250112 02:57:32.867372 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:32.868520 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_01f15888716948f8afeda9dc4bd2877d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:32.870122 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:32.870817 22931 tablet_replica.cc:331] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:32.871081 22930 ts_tablet_manager.cc:1918] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:32.871666 22931 raft_consensus.cc:2238] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:32.872179 22931 raft_consensus.cc:2267] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.874074 22931 ts_tablet_manager.cc:1905] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:32.870337 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f15888716948f8afeda9dc4bd2877d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:32.876075 21947 heartbeater.cc:346] Connected to a master server at 127.20.217.189:36049
I20250112 02:57:32.880076 22930 log.cc:1198] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:32.880687 22930 ts_tablet_manager.cc:1939] T 4e2b248224f9416faf4bd83a853ec918 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:32.823746 22987 ts_tablet_manager.cc:1905] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:32.882109 23021 raft_consensus.cc:491] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.882608 23021 raft_consensus.cc:513] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.884892 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 4e2b248224f9416faf4bd83a853ec918 (table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]) successfully deleted
I20250112 02:57:32.885607 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:32.886010 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:32.886221 22039 raft_consensus.cc:2463] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:32.887203 21691 leader_election.cc:343] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.885156 23021 leader_election.cc:290] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.887677 21690 leader_election.cc:304] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.888345 23021 raft_consensus.cc:2798] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:32.888710 23021 raft_consensus.cc:491] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:32.889035 23021 raft_consensus.cc:3054] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.891544 22930 tablet_replica.cc:331] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:32.892300 22930 raft_consensus.cc:2238] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:32.894475 22930 raft_consensus.cc:2267] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:32.894804 22143 tablet_bootstrap.cc:654] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.896277 23021 raft_consensus.cc:513] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.900285 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:32.901597 22930 ts_tablet_manager.cc:1905] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:32.901535 21691 leader_election.cc:343] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.875365 23030 raft_consensus.cc:513] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.903775 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:32.904304 22039 raft_consensus.cc:3054] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.906509 21383 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_60849e78a19643198da449366a8e6ed6" } modify_external_catalogs: true
I20250112 02:57:32.906558 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:32.907143 21383 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_60849e78a19643198da449366a8e6ed6" } modify_external_catalogs: true
W20250112 02:57:32.907840 21691 leader_election.cc:343] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:32.911549 22039 raft_consensus.cc:2463] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:32.912549 21690 leader_election.cc:304] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.902870 23021 leader_election.cc:290] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.913185 23070 raft_consensus.cc:2798] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:32.913687 23070 raft_consensus.cc:695] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:32.914521 23070 consensus_queue.cc:237] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.933945 23030 leader_election.cc:290] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:32.934583 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368af02998984cb2bee487b15b6a2b35" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:32.935303 22039 raft_consensus.cc:3054] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:32.938019 22144 tablet_bootstrap.cc:654] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:32.941973 22039 raft_consensus.cc:2463] T 368af02998984cb2bee487b15b6a2b35 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:32.942149 22930 ts_tablet_manager.cc:1918] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:32.942466 22930 log.cc:1198] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:32.942849 21690 leader_election.cc:304] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:32.943264 22930 ts_tablet_manager.cc:1939] T 15c16031f5b74cf9885fac110f445e1a P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:32.946833 22145 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:32.947216 22145 ts_tablet_manager.cc:1397] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.023s
I20250112 02:57:32.949821 22145 raft_consensus.cc:357] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.950426 22145 raft_consensus.cc:383] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:32.950731 22145 raft_consensus.cc:738] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:32.955083 22145 consensus_queue.cc:260] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.957863 22145 ts_tablet_manager.cc:1428] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250112 02:57:32.962965 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]) successfully deleted
I20250112 02:57:32.975504 21383 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet be5f2a4f6a8e4a64a7642d5a18b58a6d
I20250112 02:57:32.977030 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:32.978675 22977 raft_consensus.cc:491] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:32.979066 22977 raft_consensus.cc:513] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:32.979467 22007 tablet_service.cc:1514] Processing DeleteTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:32.980674 23055 raft_consensus.cc:2798] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:32.980862 22977 leader_election.cc:290] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:32.982246 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5f2a4f6a8e4a64a7642d5a18b58a6d already in progress: creating tablet
I20250112 02:57:32.982647 22145 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:32.982429 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:32.983460 21980 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:32.985504 23029 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 15479257910835387443) tablet 00000000000000000000000000000000 cur holder 0x7b5800071400
I20250112 02:57:32.985898 22144 tablet_bootstrap.cc:492] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:32.986533 21736 tablet_service.cc:1514] Processing DeleteTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:32.986323 22144 ts_tablet_manager.cc:1397] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.162s user 0.006s sys 0.005s
I20250112 02:57:32.987891 22145 tablet_bootstrap.cc:654] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.007922 21383 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_b36f908d431d47edbc9397abab037e2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:33.010244 22144 raft_consensus.cc:357] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.020371 22143 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:33.023379 22145 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:33.023871 22145 ts_tablet_manager.cc:1397] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.005s
I20250112 02:57:33.025892 22145 raft_consensus.cc:357] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.026718 22145 raft_consensus.cc:383] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.026993 22145 raft_consensus.cc:738] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.027599 22145 consensus_queue.cc:260] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.030126 22145 ts_tablet_manager.cc:1428] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250112 02:57:33.030893 22143 ts_tablet_manager.cc:1397] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.213s user 0.008s sys 0.005s
I20250112 02:57:33.025707 23055 raft_consensus.cc:695] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:33.031620 23055 consensus_queue.cc:237] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.035989 22145 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:33.039484 21383 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b36f908d431d47edbc9397abab037e2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:33.041005 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.041816 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.042690 22145 tablet_bootstrap.cc:654] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.045078 22143 raft_consensus.cc:357] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.045667 22143 raft_consensus.cc:383] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.045914 22143 raft_consensus.cc:738] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.062628 22144 raft_consensus.cc:383] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.068542 23089 tablet.cc:1721] T 46f8052651fa496cbb3702c17f592106 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:33.072484 22144 raft_consensus.cc:738] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.030364 22931 ts_tablet_manager.cc:1918] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:33.073159 22931 log.cc:1198] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:33.073943 22931 ts_tablet_manager.cc:1939] T 15c16031f5b74cf9885fac110f445e1a P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:33.077538 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]) successfully deleted
I20250112 02:57:33.078460 22144 consensus_queue.cc:260] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:33.085430 21978 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.085814 21978 leader_election.cc:304] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.085896 23090 tablet.cc:1721] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:33.096344 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5f2a4f6a8e4a64a7642d5a18b58a6d already in progress: creating tablet
I20250112 02:57:33.086563 23076 raft_consensus.cc:2743] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:33.093061 21978 leader_election.cc:343] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.104434 22144 ts_tablet_manager.cc:1428] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.117s user 0.005s sys 0.000s
I20250112 02:57:33.104591 21978 leader_election.cc:304] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.105259 22144 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:33.105304 23076 raft_consensus.cc:2743] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.046458 22143 consensus_queue.cc:260] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.120324 22145 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:33.120361 22143 ts_tablet_manager.cc:1428] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.077s user 0.004s sys 0.000s
I20250112 02:57:33.120790 22145 ts_tablet_manager.cc:1397] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.009s
I20250112 02:57:33.121171 22143 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:33.126629 22145 raft_consensus.cc:357] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.133239 22145 raft_consensus.cc:383] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.133574 22145 raft_consensus.cc:738] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.139281 22145 consensus_queue.cc:260] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.141251 22145 ts_tablet_manager.cc:1428] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250112 02:57:33.148474 22145 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:33.158144 22143 tablet_bootstrap.cc:654] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.170850 22145 tablet_bootstrap.cc:654] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.172312 22987 ts_tablet_manager.cc:1918] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:33.172705 22987 log.cc:1198] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/7de9ed3d1552490da7ab76037d7debc0
I20250112 02:57:33.173388 22987 ts_tablet_manager.cc:1939] T 7de9ed3d1552490da7ab76037d7debc0 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:33.179699 22144 tablet_bootstrap.cc:654] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.201176 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 235) took 1532 ms. Trace:
I20250112 02:57:33.201519 22987 rpcz_store.cc:276] 0112 02:57:31.668924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.669045 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:32.708668 (+1039623us) ts_tablet_manager.cc:1145] Deleting tablet 7de9ed3d1552490da7ab76037d7debc0
0112 02:57:32.708713 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.056381 (+347668us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:33.172081 (+115700us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:33.201110 (+ 29029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038975}
I20250112 02:57:33.202940 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 239) took 1334 ms. Trace:
I20250112 02:57:33.210860 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 7de9ed3d1552490da7ab76037d7debc0 (table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]) successfully deleted
I20250112 02:57:33.213759 22143 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:33.214295 22143 ts_tablet_manager.cc:1397] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.093s user 0.008s sys 0.002s
I20250112 02:57:33.216279 21901 raft_consensus.cc:3054] T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.216588 22143 raft_consensus.cc:357] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.216675 23076 consensus_queue.cc:1035] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:33.217245 22143 raft_consensus.cc:383] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.217484 22143 raft_consensus.cc:738] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.223347 23076 raft_consensus.cc:491] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.223744 23076 raft_consensus.cc:513] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.226291 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:33.227495 21980 leader_election.cc:343] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.228279 23076 leader_election.cc:290] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.230427 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:33.243227 21978 leader_election.cc:343] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.243600 21978 leader_election.cc:304] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.244415 23076 raft_consensus.cc:2743] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.253144 23076 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.253543 23076 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.256104 23076 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.256771 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:33.257845 21980 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.258231 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.259501 22144 tablet_bootstrap.cc:492] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:33.259765 22145 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:33.259894 22144 ts_tablet_manager.cc:1397] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.000s
I20250112 02:57:33.260134 22145 ts_tablet_manager.cc:1397] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.112s user 0.021s sys 0.005s
I20250112 02:57:33.262310 22145 raft_consensus.cc:357] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.262928 22145 raft_consensus.cc:383] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.263157 22145 raft_consensus.cc:738] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.263669 22145 consensus_queue.cc:260] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:33.264717 21978 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.265090 21978 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.203275 22987 rpcz_store.cc:276] 0112 02:57:31.868494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.868649 (+ 155us) service_pool.cc:225] Handling call
0112 02:57:33.202575 (+1333926us) ts_tablet_manager.cc:1145] Deleting tablet edfa495bfefc46b998c2f870d9014900
0112 02:57:33.202616 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.202887 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1333281}
W20250112 02:57:33.267161 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet edfa495bfefc46b998c2f870d9014900 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edfa495bfefc46b998c2f870d9014900 already in progress: creating tablet
I20250112 02:57:33.268476 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 240) took 1396 ms. Trace:
I20250112 02:57:33.268821 22987 rpcz_store.cc:276] 0112 02:57:31.872509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.872841 (+ 332us) service_pool.cc:225] Handling call
0112 02:57:33.268106 (+1395265us) ts_tablet_manager.cc:1145] Deleting tablet f3da552a973945e284a386304f182777
0112 02:57:33.268150 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.268418 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1394723}
W20250112 02:57:33.269910 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet f3da552a973945e284a386304f182777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3da552a973945e284a386304f182777 already in progress: creating tablet
I20250112 02:57:33.280053 22977 raft_consensus.cc:2743] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.280964 22977 raft_consensus.cc:491] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.281322 22977 raft_consensus.cc:513] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.261986 22144 raft_consensus.cc:357] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.282573 22144 raft_consensus.cc:383] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.282857 22144 raft_consensus.cc:738] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.283159 22977 leader_election.cc:290] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.283661 22144 consensus_queue.cc:260] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.288154 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:33.291360 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 244) took 1069 ms. Trace:
W20250112 02:57:33.291582 21980 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:33.292234 21980 consensus_peers.cc:487] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:33.291779 22987 rpcz_store.cc:276] 0112 02:57:32.221560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.221695 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:33.290905 (+1069210us) ts_tablet_manager.cc:1145] Deleting tablet 9c31714d18414800845671d783ac8173
0112 02:57:33.290958 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.291288 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038801}
I20250112 02:57:33.293021 22987 tablet_replica.cc:331] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:33.293936 22987 raft_consensus.cc:2238] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:33.218004 22143 consensus_queue.cc:260] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.302850 22143 ts_tablet_manager.cc:1428] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.088s user 0.005s sys 0.000s
W20250112 02:57:33.303848 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 9c31714d18414800845671d783ac8173 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9c31714d18414800845671d783ac8173 already in progress: creating tablet
I20250112 02:57:33.304714 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.305639 22145 ts_tablet_manager.cc:1428] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.045s user 0.006s sys 0.001s
W20250112 02:57:33.306682 21978 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.307058 21978 leader_election.cc:304] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.307909 23076 raft_consensus.cc:2743] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.309607 21902 raft_consensus.cc:3054] T e21a0e4b5ebc426383cf332f46492b80 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.330618 22143 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:33.342605 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 1db9ff39ff2341199549155a2db0aaba (DEFAULT_TABLE table=default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:33.342929 23097 log.cc:926] Time spent T 67260685978247898ad224128ed8964f P cca160b7cffb4ae287ea077e7352ffc9: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250112 02:57:33.343593 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db9ff39ff2341199549155a2db0aaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.286437 22144 ts_tablet_manager.cc:1428] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250112 02:57:33.347361 22144 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:33.355013 22145 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:33.294421 22987 raft_consensus.cc:2267] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:33.378705 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 1db9ff39ff2341199549155a2db0aaba (DEFAULT_TABLE table=default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.379738 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db9ff39ff2341199549155a2db0aaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.379761 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 1db9ff39ff2341199549155a2db0aaba (DEFAULT_TABLE table=default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.380905 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1db9ff39ff2341199549155a2db0aaba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.385016 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8d7d76e5184d668e4add9810fab7df (DEFAULT_TABLE table=default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.385175 21879 tablet_service.cc:1467] Processing CreateTablet for tablet c01590521b6e4ddc9d7fcc7142d49930 (DEFAULT_TABLE table=default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.385991 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d7d76e5184d668e4add9810fab7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.386090 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01590521b6e4ddc9d7fcc7142d49930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.389797 22012 tablet_service.cc:1467] Processing CreateTablet for tablet c01590521b6e4ddc9d7fcc7142d49930 (DEFAULT_TABLE table=default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.390938 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01590521b6e4ddc9d7fcc7142d49930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.399410 22143 tablet_bootstrap.cc:654] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.400494 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8d7d76e5184d668e4add9810fab7df (DEFAULT_TABLE table=default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.400676 21732 tablet_service.cc:1467] Processing CreateTablet for tablet c01590521b6e4ddc9d7fcc7142d49930 (DEFAULT_TABLE table=default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.401569 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d7d76e5184d668e4add9810fab7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.402066 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c01590521b6e4ddc9d7fcc7142d49930. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.402279 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 8a8d7d76e5184d668e4add9810fab7df (DEFAULT_TABLE table=default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.403344 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a8d7d76e5184d668e4add9810fab7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.422441 22987 ts_tablet_manager.cc:1905] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:33.436506 22144 tablet_bootstrap.cc:654] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.457262 21901 raft_consensus.cc:3054] T 2cbf5e2a26474c98b9cb48b5bbdceafb P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.463913 22145 tablet_bootstrap.cc:654] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.468822 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3fd2d14dbf4bc98e42a04e33c66348 (DEFAULT_TABLE table=default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.469916 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3fd2d14dbf4bc98e42a04e33c66348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.469877 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3fd2d14dbf4bc98e42a04e33c66348 (DEFAULT_TABLE table=default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.470788 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3fd2d14dbf4bc98e42a04e33c66348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.472154 21880 tablet_service.cc:1467] Processing CreateTablet for tablet b32e6564d4dd4d1b9b25a14266b3dc04 (DEFAULT_TABLE table=default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.472456 21727 tablet_service.cc:1467] Processing CreateTablet for tablet b32e6564d4dd4d1b9b25a14266b3dc04 (DEFAULT_TABLE table=default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.473028 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32e6564d4dd4d1b9b25a14266b3dc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.473376 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32e6564d4dd4d1b9b25a14266b3dc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.487632 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 5c3fd2d14dbf4bc98e42a04e33c66348 (DEFAULT_TABLE table=default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.488837 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c3fd2d14dbf4bc98e42a04e33c66348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.496469 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8 (DEFAULT_TABLE table=default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.497458 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.498063 22019 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8 (DEFAULT_TABLE table=default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.498935 22143 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:33.499109 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.499315 22143 ts_tablet_manager.cc:1397] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.000s
I20250112 02:57:33.501473 22143 raft_consensus.cc:357] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.502080 22143 raft_consensus.cc:383] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.508827 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8 (DEFAULT_TABLE table=default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.512516 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.499100 21732 tablet_service.cc:1467] Processing CreateTablet for tablet b4aff0623f504bf09011ca6de28a3b82 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.517534 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aff0623f504bf09011ca6de28a3b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.520411 22014 tablet_service.cc:1467] Processing CreateTablet for tablet b32e6564d4dd4d1b9b25a14266b3dc04 (DEFAULT_TABLE table=default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.530440 22145 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:33.530849 22145 ts_tablet_manager.cc:1397] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.176s user 0.008s sys 0.003s
I20250112 02:57:33.533450 22145 raft_consensus.cc:357] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.534065 22145 raft_consensus.cc:383] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.534302 22145 raft_consensus.cc:738] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.534934 22145 consensus_queue.cc:260] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.537153 22019 tablet_service.cc:1467] Processing CreateTablet for tablet b4aff0623f504bf09011ca6de28a3b82 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.541625 22144 tablet_bootstrap.cc:492] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:33.542044 22144 ts_tablet_manager.cc:1397] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.195s user 0.008s sys 0.002s
I20250112 02:57:33.543761 21876 tablet_service.cc:1467] Processing CreateTablet for tablet b4aff0623f504bf09011ca6de28a3b82 (DEFAULT_TABLE table=default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.544667 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aff0623f504bf09011ca6de28a3b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.502316 22143 raft_consensus.cc:738] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.545284 22143 consensus_queue.cc:260] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.547117 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbce2d946cd47398a3d0a1f72ca1678 (DEFAULT_TABLE table=default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.547684 22143 ts_tablet_manager.cc:1428] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20250112 02:57:33.547989 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbce2d946cd47398a3d0a1f72ca1678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.548509 22143 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:33.537679 22145 ts_tablet_manager.cc:1428] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250112 02:57:33.538012 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32e6564d4dd4d1b9b25a14266b3dc04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.556087 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4aff0623f504bf09011ca6de28a3b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.571362 21392 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:33.574043 21392 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:33.544261 22144 raft_consensus.cc:357] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.574645 22144 raft_consensus.cc:383] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.575009 22144 raft_consensus.cc:738] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.575672 22144 consensus_queue.cc:260] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.578276 22144 ts_tablet_manager.cc:1428] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20250112 02:57:33.557441 22145 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:33.558255 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbce2d946cd47398a3d0a1f72ca1678 (DEFAULT_TABLE table=default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.583626 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbce2d946cd47398a3d0a1f72ca1678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.596536 23100 raft_consensus.cc:491] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.597020 23100 raft_consensus.cc:513] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.599926 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:33.601217 21978 leader_election.cc:343] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.601778 23100 leader_election.cc:290] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.603642 21902 raft_consensus.cc:3054] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.603482 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:33.604749 21980 leader_election.cc:343] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.605106 21980 leader_election.cc:304] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.606175 22144 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:33.606616 23100 raft_consensus.cc:2743] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.609841 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbce2d946cd47398a3d0a1f72ca1678 (DEFAULT_TABLE table=default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:33.612717 22145 tablet_bootstrap.cc:654] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.622680 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc0757376c3411a9fcbc84ec411befd. Latest consensus state: current_term: 3 leader_uuid: "bcc0757376c3411a9fcbc84ec411befd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:33.623369 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:33.623883 23049 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:33.624424 23049 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:33.649149 22144 tablet_bootstrap.cc:654] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.652345 23049 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:33.634737 22143 tablet_bootstrap.cc:654] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.635862 23055 raft_consensus.cc:491] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.658399 23055 raft_consensus.cc:513] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.660298 23055 leader_election.cc:290] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.651544 23022 raft_consensus.cc:491] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.661592 23022 raft_consensus.cc:513] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.662568 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:33.663236 23022 leader_election.cc:290] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:33.664763 21690 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: a2ce633c5b3b4cbea17c171564fa72d6
W20250112 02:57:33.665515 21691 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.665810 21691 leader_election.cc:304] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:33.666939 23055 raft_consensus.cc:2743] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.634872 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbce2d946cd47398a3d0a1f72ca1678. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:33.668483 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:33.669242 22038 raft_consensus.cc:2463] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:33.670512 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:33.673153 22145 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:33.673520 22145 ts_tablet_manager.cc:1397] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.116s user 0.009s sys 0.000s
I20250112 02:57:33.673949 21392 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:33.674563 21392 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:33.676661 22145 raft_consensus.cc:357] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.677280 22145 raft_consensus.cc:383] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.677527 22145 raft_consensus.cc:738] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.672405 21690 leader_election.cc:304] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:33.682929 21691 leader_election.cc:343] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.684981 23022 raft_consensus.cc:2798] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:33.685328 23022 raft_consensus.cc:491] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:33.685631 23022 raft_consensus.cc:3054] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.680675 22143 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:33.686933 22143 ts_tablet_manager.cc:1397] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.000s
I20250112 02:57:33.689123 22143 raft_consensus.cc:357] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.689798 22143 raft_consensus.cc:383] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.690037 22143 raft_consensus.cc:738] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.671679 22987 ts_tablet_manager.cc:1918] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:33.690939 22987 log.cc:1198] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:33.691568 22987 ts_tablet_manager.cc:1939] T 4e2b248224f9416faf4bd83a853ec918 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:33.692031 22144 tablet_bootstrap.cc:492] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:33.692405 22144 ts_tablet_manager.cc:1397] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.000s
I20250112 02:57:33.678082 22145 consensus_queue.cc:260] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.699890 23022 raft_consensus.cc:513] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.704450 23109 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 02:57:33.717190 22145 ts_tablet_manager.cc:1428] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20250112 02:57:33.694417 22144 raft_consensus.cc:357] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.731149 22144 raft_consensus.cc:383] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:33.731424 22144 raft_consensus.cc:738] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:33.739059 23100 raft_consensus.cc:491] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.739523 23100 raft_consensus.cc:513] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.744418 23100 leader_election.cc:290] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.744406 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:33.745405 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:33.746018 21980 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:33.746627 21978 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.746979 21978 leader_election.cc:304] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.747869 23100 raft_consensus.cc:2743] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.749619 23100 raft_consensus.cc:491] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.749306 23055 consensus_queue.cc:1035] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:33.749966 23100 raft_consensus.cc:513] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.751605 23100 leader_election.cc:290] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.752388 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 246) took 1190 ms. Trace:
I20250112 02:57:33.752830 22987 rpcz_store.cc:276] 0112 02:57:32.562275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.562410 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:33.292842 (+730432us) ts_tablet_manager.cc:1145] Deleting tablet 4e2b248224f9416faf4bd83a853ec918
0112 02:57:33.292892 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.461537 (+168645us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:33.671449 (+209912us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:33.752002 (+ 80553us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400131428. stack: 00007fe304724853 00007fe304724750 00007fe304299584 00000000004fcd24 00000000004fccb9 00007fe30ba3ba81 00007fe30bac2b5c 00007fe30c6d4e6b 00007fe30c704ca0 00007fe30c6ebe6d 00007fe30c6ebe19 00007fe30c6ebda9 00007fe30c6ebd71 00007fe30c6eaf9d 000000000059ea04 000000000059e839
0112 02:57:33.752313 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51083136,"tablet-delete.queue_time_us":716917}
I20250112 02:57:33.753554 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:33.754432 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 4e2b248224f9416faf4bd83a853ec918 (table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]) successfully deleted
I20250112 02:57:33.754969 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 245) took 1214 ms. Trace:
W20250112 02:57:33.755468 21980 leader_election.cc:343] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.755379 22987 rpcz_store.cc:276] 0112 02:57:32.540193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.549631 (+ 9438us) service_pool.cc:225] Handling call
0112 02:57:33.754545 (+1204914us) ts_tablet_manager.cc:1145] Deleting tablet 922b22f0b1284bc4a732371a61cb1ed4
0112 02:57:33.754598 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.754906 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1175002}
W20250112 02:57:33.756804 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 922b22f0b1284bc4a732371a61cb1ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 922b22f0b1284bc4a732371a61cb1ed4 already in progress: creating tablet
I20250112 02:57:33.778872 22145 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:33.775131 22143 consensus_queue.cc:260] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.784716 22143 ts_tablet_manager.cc:1428] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.098s user 0.004s sys 0.001s
I20250112 02:57:33.785528 22143 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:33.789175 22144 consensus_queue.cc:260] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.790870 21900 raft_consensus.cc:3054] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.791083 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.791612 22144 ts_tablet_manager.cc:1428] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.099s user 0.005s sys 0.001s
I20250112 02:57:33.792011 22996 raft_consensus.cc:491] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2770701ca54288956df9d910d7d3d2)
I20250112 02:57:33.792408 22144 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:33.792443 22996 raft_consensus.cc:513] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:33.795693 21978 leader_election.cc:343] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.796072 21978 leader_election.cc:304] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.801128 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.803336 22996 leader_election.cc:290] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.804697 23022 leader_election.cc:290] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.806953 22987 tablet_replica.cc:331] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:33.807958 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:33.810353 21691 consensus_peers.cc:487] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:33.811043 21691 leader_election.cc:343] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.820606 22865 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" instance_seqno: 1736650630390360) as {username='slave'} at 127.20.217.130:54033; Asking this server to re-register.
W20250112 02:57:33.824896 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 750246f0a0c549229d4e7a87cd6ddfd9 consensus state has no leader
I20250112 02:57:33.830889 21947 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:33.833427 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:33.837154 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:33.839799 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:33.840333 22045 raft_consensus.cc:3054] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.839948 21836 leader_election.cc:304] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:33.834990 23035 raft_consensus.cc:2743] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.843341 22996 raft_consensus.cc:2743] T e8f7301bed6c4d98a80c12b70073e62f P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:33.850077 22987 raft_consensus.cc:2238] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:33.850207 22045 raft_consensus.cc:2463] T 8b6efb19f0cc4a96b957ec34c27ab836 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:33.851600 22987 raft_consensus.cc:2267] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:33.851657 23126 consensus_queue.cc:1035] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:33.853792 22987 ts_tablet_manager.cc:1905] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:33.856176 21690 leader_election.cc:304] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.857322 23124 raft_consensus.cc:2798] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:33.857739 23124 raft_consensus.cc:695] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:33.858675 23124 consensus_queue.cc:237] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.870415 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:33.872473 22840 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:33.873695 23049 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:33.873895 22865 ts_manager.cc:194] Registered new tserver with Master: cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:33.876173 23022 raft_consensus.cc:491] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.876580 23022 raft_consensus.cc:513] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.879031 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:33.880273 21691 leader_election.cc:343] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.881317 23022 leader_election.cc:290] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.881873 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:33.882447 22038 raft_consensus.cc:2463] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:33.883446 21690 leader_election.cc:304] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.884409 23022 raft_consensus.cc:2798] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:33.884766 23022 raft_consensus.cc:491] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:33.885113 23022 raft_consensus.cc:3054] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:33.888542 22145 tablet_bootstrap.cc:654] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.891944 22144 tablet_bootstrap.cc:654] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.900477 22143 tablet_bootstrap.cc:654] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:33.901192 23076 raft_consensus.cc:491] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.901780 23076 raft_consensus.cc:513] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.904623 23076 leader_election.cc:290] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:33.906822 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:33.907999 21978 leader_election.cc:343] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.932580 23022 raft_consensus.cc:513] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.905473 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:33.947578 21980 leader_election.cc:343] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.947976 21980 leader_election.cc:304] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.948665 23035 raft_consensus.cc:2743] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:33.950059 22865 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" instance_seqno: 1736650632306429) as {username='slave'} at 127.20.217.131:51299; Asking this server to re-register.
I20250112 02:57:33.955200 22090 heartbeater.cc:463] Registering TS with master...
W20250112 02:57:33.963303 21691 consensus_peers.cc:487] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:33.965667 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:33.965802 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:33.966245 22044 raft_consensus.cc:3054] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:33.966846 21691 leader_election.cc:343] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:33.973147 22044 raft_consensus.cc:2463] T e4a9bc28dc5942608fb7224f9f058e27 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:33.973966 21690 leader_election.cc:304] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:33.974682 23030 raft_consensus.cc:2798] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:33.974751 22996 raft_consensus.cc:491] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.975281 23030 raft_consensus.cc:695] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:33.975210 22996 raft_consensus.cc:513] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.976944 22996 leader_election.cc:290] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.978480 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:33.980280 22143 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:33.976245 23030 consensus_queue.cc:237] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.980684 22143 ts_tablet_manager.cc:1397] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.195s user 0.007s sys 0.003s
I20250112 02:57:33.984321 23022 leader_election.cc:290] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:33.987531 22864 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" instance_seqno: 1736650627879077) as {username='slave'} at 127.20.217.129:58767; Asking this server to re-register.
I20250112 02:57:33.991003 21802 heartbeater.cc:463] Registering TS with master...
I20250112 02:57:33.995323 23076 raft_consensus.cc:491] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:33.995698 23076 raft_consensus.cc:513] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:33.997453 23076 leader_election.cc:290] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:33.999099 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:34.000321 21980 leader_election.cc:343] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.001549 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:34.002763 21978 leader_election.cc:343] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.003103 21978 leader_election.cc:304] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.003955 23076 raft_consensus.cc:2743] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.005188 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:34.006367 21836 leader_election.cc:304] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:34.007421 22996 raft_consensus.cc:2743] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.014436 22987 ts_tablet_manager.cc:1918] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250112 02:57:34.010818 22143 raft_consensus.cc:357] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.019454 22143 raft_consensus.cc:383] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.019694 22143 raft_consensus.cc:738] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.020244 22143 consensus_queue.cc:260] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.022568 22143 ts_tablet_manager.cc:1428] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.042s user 0.006s sys 0.000s
I20250112 02:57:34.014838 22987 log.cc:1198] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:34.027274 22987 ts_tablet_manager.cc:1939] T 15c16031f5b74cf9885fac110f445e1a P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:34.028585 22865 ts_manager.cc:194] Registered new tserver with Master: e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:34.034603 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.727s user 0.027s sys 0.000s
I20250112 02:57:34.039410 22144 tablet_bootstrap.cc:492] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:34.039796 22144 ts_tablet_manager.cc:1397] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.248s user 0.009s sys 0.000s
I20250112 02:57:34.041095 22864 ts_manager.cc:194] Registered new tserver with Master: 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:34.042286 22144 raft_consensus.cc:357] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.042963 22144 raft_consensus.cc:383] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.043226 22144 raft_consensus.cc:738] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.043804 22144 consensus_queue.cc:260] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.055339 22145 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.055792 22145 ts_tablet_manager.cc:1397] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.277s user 0.022s sys 0.027s
I20250112 02:57:34.060262 22145 raft_consensus.cc:357] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.060942 22145 raft_consensus.cc:383] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.061337 22145 raft_consensus.cc:738] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.062045 22145 consensus_queue.cc:260] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.063817 22144 ts_tablet_manager.cc:1428] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20250112 02:57:34.065682 22145 ts_tablet_manager.cc:1428] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250112 02:57:34.087203 22143 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.089458 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 247) took 1222 ms. Trace:
I20250112 02:57:34.091796 22145 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.098819 22144 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:34.104525 22144 tablet_bootstrap.cc:654] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.089798 22987 rpcz_store.cc:276] 0112 02:57:32.867128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.867225 (+ 97us) service_pool.cc:225] Handling call
0112 02:57:33.806775 (+939550us) ts_tablet_manager.cc:1145] Deleting tablet 15c16031f5b74cf9885fac110f445e1a
0112 02:57:33.806827 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:33.851330 (+ 44503us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b5400110b54. stack: 00007fe304724853 00007fe304724750 00007fe304299584 00000000004fcd24 00000000004fccb9 00007fe30b6246d1 00007fe30b6a7352 00007fe30bac41ac 00007fe30c6d4d5a 00007fe30c704ca0 00007fe30c6ebe6d 00007fe30c6ebe19 00007fe30c6ebda9 00007fe30c6ebd71 00007fe30c6eaf9d 000000000059ea04
0112 02:57:33.954667 (+103337us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:34.013893 (+ 59226us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:34.089397 (+ 75504us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96914176,"tablet-delete.queue_time_us":939017}
I20250112 02:57:34.107239 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 15c16031f5b74cf9885fac110f445e1a (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]) successfully deleted
I20250112 02:57:34.113282 21900 raft_consensus.cc:3054] T d6ca275360ec41ceb0f7ef9bfaedf647 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.116955 22145 tablet_bootstrap.cc:654] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.128815 22143 tablet_bootstrap.cc:654] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.131762 22987 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 248) took 1147 ms. Trace:
I20250112 02:57:34.132668 22987 rpcz_store.cc:276] 0112 02:57:32.984269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.984360 (+ 91us) service_pool.cc:225] Handling call
0112 02:57:34.114448 (+1130088us) ts_tablet_manager.cc:1145] Deleting tablet be5f2a4f6a8e4a64a7642d5a18b58a6d
0112 02:57:34.131441 (+ 16993us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:34.131707 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127530}
W20250112 02:57:34.133946 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be5f2a4f6a8e4a64a7642d5a18b58a6d already in progress: creating tablet
I20250112 02:57:34.164384 21902 raft_consensus.cc:3054] T 368af02998984cb2bee487b15b6a2b35 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.171836 23030 consensus_queue.cc:1035] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:34.178397 22145 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.178804 22145 ts_tablet_manager.cc:1397] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.087s user 0.013s sys 0.000s
I20250112 02:57:34.180820 22145 raft_consensus.cc:357] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.181429 22145 raft_consensus.cc:383] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.181660 22145 raft_consensus.cc:738] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.187268 22145 consensus_queue.cc:260] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.189664 22145 ts_tablet_manager.cc:1428] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250112 02:57:34.200500 22145 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.207255 22143 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:34.207676 22143 ts_tablet_manager.cc:1397] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.000s
I20250112 02:57:34.217608 22145 tablet_bootstrap.cc:654] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.221585 23126 consensus_queue.cc:1035] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:34.210178 22143 raft_consensus.cc:357] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.223295 22143 raft_consensus.cc:383] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.223512 22143 raft_consensus.cc:738] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.232800 22145 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.233197 22145 ts_tablet_manager.cc:1397] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.004s
I20250112 02:57:34.235416 22143 consensus_queue.cc:260] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.236032 22145 raft_consensus.cc:357] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.236642 22145 raft_consensus.cc:383] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.236866 22145 raft_consensus.cc:738] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.237515 22145 consensus_queue.cc:260] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.239900 22145 ts_tablet_manager.cc:1428] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250112 02:57:34.242064 22145 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.247320 22145 tablet_bootstrap.cc:654] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.253744 22143 ts_tablet_manager.cc:1428] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250112 02:57:34.275563 21730 tablet_service.cc:1467] Processing CreateTablet for tablet ee7967f6294b4cfeac9e97db8de41e11 (DEFAULT_TABLE table=default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.276567 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7967f6294b4cfeac9e97db8de41e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.279505 22007 tablet_service.cc:1467] Processing CreateTablet for tablet ee7967f6294b4cfeac9e97db8de41e11 (DEFAULT_TABLE table=default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.280138 21872 tablet_service.cc:1467] Processing CreateTablet for tablet ee7967f6294b4cfeac9e97db8de41e11 (DEFAULT_TABLE table=default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.280354 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7967f6294b4cfeac9e97db8de41e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.280941 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee7967f6294b4cfeac9e97db8de41e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.284348 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c28220edf4598af19ba21e3e7b7d3 (DEFAULT_TABLE table=default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.285132 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6c28220edf4598af19ba21e3e7b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.287955 22144 tablet_bootstrap.cc:492] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:34.288353 22144 ts_tablet_manager.cc:1397] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.190s user 0.004s sys 0.010s
I20250112 02:57:34.288520 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c28220edf4598af19ba21e3e7b7d3 (DEFAULT_TABLE table=default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.289628 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6c28220edf4598af19ba21e3e7b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.290052 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6c28220edf4598af19ba21e3e7b7d3 (DEFAULT_TABLE table=default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.291005 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f6c28220edf4598af19ba21e3e7b7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.291136 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 7634a1d8de9b495599b83f15eaa91c89 (DEFAULT_TABLE table=default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.291913 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7634a1d8de9b495599b83f15eaa91c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.294795 22143 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.296730 22144 raft_consensus.cc:357] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.308670 22144 raft_consensus.cc:383] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.308933 22144 raft_consensus.cc:738] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.309525 22144 consensus_queue.cc:260] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.320575 22144 ts_tablet_manager.cc:1428] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.032s user 0.004s sys 0.003s
I20250112 02:57:34.321359 22144 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:34.336792 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 56c7f07da89c4994a57c23ebdd8061bf (DEFAULT_TABLE table=default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.337781 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56c7f07da89c4994a57c23ebdd8061bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.337606 21736 tablet_service.cc:1467] Processing CreateTablet for tablet 56c7f07da89c4994a57c23ebdd8061bf (DEFAULT_TABLE table=default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.338478 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56c7f07da89c4994a57c23ebdd8061bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.349620 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 7634a1d8de9b495599b83f15eaa91c89 (DEFAULT_TABLE table=default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.354203 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7634a1d8de9b495599b83f15eaa91c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.359195 22144 tablet_bootstrap.cc:654] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.375413 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 7634a1d8de9b495599b83f15eaa91c89 (DEFAULT_TABLE table=default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.376359 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7634a1d8de9b495599b83f15eaa91c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.388983 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 56c7f07da89c4994a57c23ebdd8061bf (DEFAULT_TABLE table=default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.389361 21726 tablet_service.cc:1467] Processing CreateTablet for tablet d1e0548276474b68919cb10fbcfcdf04 (DEFAULT_TABLE table=default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.390084 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56c7f07da89c4994a57c23ebdd8061bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.390239 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e0548276474b68919cb10fbcfcdf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.390184 21880 tablet_service.cc:1467] Processing CreateTablet for tablet d1e0548276474b68919cb10fbcfcdf04 (DEFAULT_TABLE table=default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.391016 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e0548276474b68919cb10fbcfcdf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.392776 21877 tablet_service.cc:1467] Processing CreateTablet for tablet da54d9dbab2f416785ec0379095bcf3e (DEFAULT_TABLE table=default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.393561 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da54d9dbab2f416785ec0379095bcf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.395046 21727 tablet_service.cc:1467] Processing CreateTablet for tablet da54d9dbab2f416785ec0379095bcf3e (DEFAULT_TABLE table=default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.395588 22012 tablet_service.cc:1467] Processing CreateTablet for tablet d1e0548276474b68919cb10fbcfcdf04 (DEFAULT_TABLE table=default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.395867 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da54d9dbab2f416785ec0379095bcf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.396526 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1e0548276474b68919cb10fbcfcdf04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.408125 22144 tablet_bootstrap.cc:492] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:34.408512 22144 ts_tablet_manager.cc:1397] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.087s user 0.008s sys 0.000s
I20250112 02:57:34.411132 22144 raft_consensus.cc:357] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.417388 22144 raft_consensus.cc:383] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.417635 22144 raft_consensus.cc:738] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.368346 22143 tablet_bootstrap.cc:654] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.444319 22019 tablet_service.cc:1467] Processing CreateTablet for tablet da54d9dbab2f416785ec0379095bcf3e (DEFAULT_TABLE table=default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.445319 21879 tablet_service.cc:1467] Processing CreateTablet for tablet af0c74dd3303445b9e9aef1f2d7af62a (DEFAULT_TABLE table=default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.446237 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0c74dd3303445b9e9aef1f2d7af62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.451274 21727 tablet_service.cc:1467] Processing CreateTablet for tablet af0c74dd3303445b9e9aef1f2d7af62a (DEFAULT_TABLE table=default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.452183 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0c74dd3303445b9e9aef1f2d7af62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.456065 22145 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.456542 22145 ts_tablet_manager.cc:1397] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.215s user 0.009s sys 0.000s
I20250112 02:57:34.458804 23030 raft_consensus.cc:491] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.459221 23030 raft_consensus.cc:513] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.461004 23030 leader_election.cc:290] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.461627 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:34.463095 21691 leader_election.cc:343] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.419569 22144 consensus_queue.cc:260] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.466490 22144 ts_tablet_manager.cc:1428] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.057s user 0.004s sys 0.000s
I20250112 02:57:34.470448 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:34.471136 22038 raft_consensus.cc:2463] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:34.472718 21690 leader_election.cc:304] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.473577 23030 raft_consensus.cc:2798] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:34.473898 23030 raft_consensus.cc:491] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:34.474215 23030 raft_consensus.cc:3054] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.476954 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da54d9dbab2f416785ec0379095bcf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.458863 22145 raft_consensus.cc:357] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.481104 23030 raft_consensus.cc:513] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.483829 22145 raft_consensus.cc:383] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.484684 23030 leader_election.cc:290] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.484297 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:34.485215 22044 raft_consensus.cc:3054] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.491118 22044 raft_consensus.cc:2463] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:34.492002 21690 leader_election.cc:304] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:34.492838 23030 raft_consensus.cc:2798] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:34.493191 22145 raft_consensus.cc:738] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.493769 23030 raft_consensus.cc:695] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:34.499795 22996 raft_consensus.cc:491] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.500279 22996 raft_consensus.cc:513] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.502214 22996 leader_election.cc:290] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.505519 23076 raft_consensus.cc:491] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.505824 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:34.505939 23076 raft_consensus.cc:513] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.507692 23076 leader_election.cc:290] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:34.507851 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 929430298434492cbab3d197e1083400 (DEFAULT_TABLE table=default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.509048 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 929430298434492cbab3d197e1083400 (DEFAULT_TABLE table=default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.509214 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 67236f9e5d11467486e81393dd01e0df (DEFAULT_TABLE table=default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.504325 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118ef6d7c16941598177b92a94c4b472" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:34.510006 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929430298434492cbab3d197e1083400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.509912 22014 tablet_service.cc:1467] Processing CreateTablet for tablet af0c74dd3303445b9e9aef1f2d7af62a (DEFAULT_TABLE table=default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.510211 23030 consensus_queue.cc:237] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.510816 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af0c74dd3303445b9e9aef1f2d7af62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.507009 22143 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:34.511790 22143 ts_tablet_manager.cc:1397] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.257s user 0.011s sys 0.001s
I20250112 02:57:34.513811 22143 raft_consensus.cc:357] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.515093 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 67236f9e5d11467486e81393dd01e0df (DEFAULT_TABLE table=default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.516013 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67236f9e5d11467486e81393dd01e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.516616 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:34.518111 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:34.494581 22145 consensus_queue.cc:260] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.519091 21835 leader_election.cc:304] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:34.518827 21386 catalog_manager.cc:5526] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
I20250112 02:57:34.520134 22145 ts_tablet_manager.cc:1428] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.063s user 0.004s sys 0.000s
I20250112 02:57:34.520648 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:34.520905 22145 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.521088 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]). Expected version 1 got 0
I20250112 02:57:34.520998 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 67236f9e5d11467486e81393dd01e0df (DEFAULT_TABLE table=default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.521581 22996 raft_consensus.cc:2743] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.522058 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67236f9e5d11467486e81393dd01e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.522639 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 929430298434492cbab3d197e1083400 (DEFAULT_TABLE table=default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:34.523563 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929430298434492cbab3d197e1083400. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:34.527940 21978 leader_election.cc:343] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.514384 22143 raft_consensus.cc:383] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.534123 22143 raft_consensus.cc:738] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.534685 22143 consensus_queue.cc:260] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.537020 22143 ts_tablet_manager.cc:1428] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.025s user 0.001s sys 0.003s
I20250112 02:57:34.537781 22143 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.545853 22145 tablet_bootstrap.cc:654] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.521701 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:34.547200 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]). Expected version 1 got 0
W20250112 02:57:34.548213 21385 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82825c2061d74cafa814912bec005654 consensus state has no leader
I20250112 02:57:34.548687 21385 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 283) took 2220 ms. Trace:
I20250112 02:57:34.548992 21385 rpcz_store.cc:276] 0112 02:57:32.327906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.541775 (+213869us) service_pool.cc:225] Handling call
0112 02:57:32.556669 (+ 14894us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:32.558281 (+ 1612us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:32.565676 (+ 7395us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:32.566004 (+ 328us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:32.588872 (+ 22868us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:34.548632 (+1959760us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:32.567479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:32.567877 (+ 398us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:32.567905 (+ 28us) write_op.cc:435] Acquired schema lock
0112 02:57:32.567918 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:32.568195 (+ 277us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:32.568238 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:32.568263 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:32.568419 (+ 156us) tablet.cc:698] Row locks acquired
0112 02:57:32.568434 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:32.568546 (+ 112us) write_op.cc:270] Start()
0112 02:57:32.568652 (+ 106us) write_op.cc:276] Timestamp: P: 1736650652568626 usec, L: 0
0112 02:57:32.568670 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:32.569199 (+ 529us) log.cc:844] Serialized 405 byte log entry
0112 02:57:32.576772 (+ 7573us) op_driver.cc:464] REPLICATION: finished
0112 02:57:32.577601 (+ 829us) write_op.cc:301] APPLY: starting
0112 02:57:32.577650 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:32.577743 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:32.577843 (+ 100us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:32.578007 (+ 164us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:32.578138 (+ 131us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:32.578178 (+ 40us) write_op.cc:312] APPLY: finished
0112 02:57:32.578346 (+ 168us) log.cc:844] Serialized 23 byte log entry
0112 02:57:32.578489 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.580887 (+ 2398us) write_op.cc:454] Released schema lock
0112 02:57:32.580938 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:32.581145 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.581167 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":688,"apply.run_cpu_time_us":1217,"apply.run_wall_time_us":3514,"num_ops":1,"prepare.queue_time_us":96,"prepare.run_cpu_time_us":2003,"prepare.run_wall_time_us":2002,"raft.queue_time_us":1322,"raft.run_cpu_time_us":631,"raft.run_wall_time_us":631,"replication_time_us":7997}]]}
W20250112 02:57:34.552433 21385 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 37) took 1020 ms (client timeout 1000 ms). Trace:
W20250112 02:57:34.552702 21385 rpcz_store.cc:269] 0112 02:57:33.532083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:34.552297 (+1020214us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:34.552383 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250112 02:57:34.467289 22144 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:34.584784 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:34.585726 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:34.586933 21980 leader_election.cc:343] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.587328 21980 leader_election.cc:304] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.588239 23076 raft_consensus.cc:2743] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:34.591219 21691 leader_election.cc:343] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.595400 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67236f9e5d11467486e81393dd01e0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.601532 21383 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:34.602547 21383 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:34.603474 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929430298434492cbab3d197e1083400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:34.605545 22143 tablet_bootstrap.cc:654] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.630344 22143 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:34.633898 22883 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56208 (request call id 0) took 3937 ms. Trace:
I20250112 02:57:34.634312 22145 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.634744 22145 ts_tablet_manager.cc:1397] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.114s user 0.008s sys 0.002s
I20250112 02:57:34.637063 22145 raft_consensus.cc:357] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.637660 22145 raft_consensus.cc:383] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.637881 22145 raft_consensus.cc:738] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.638607 22143 ts_tablet_manager.cc:1397] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.005s
I20250112 02:57:34.641162 22145 consensus_queue.cc:260] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.650264 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:34.640663 22143 raft_consensus.cc:357] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.651093 22143 raft_consensus.cc:383] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.651329 22143 raft_consensus.cc:738] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.652410 22145 ts_tablet_manager.cc:1428] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20250112 02:57:34.653359 22145 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.663300 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:34.664073 22143 consensus_queue.cc:260] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.665073 21385 catalog_manager.cc:5526] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:34.666342 21385 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]). Expected version 1 got 0
I20250112 02:57:34.669041 21385 catalog_manager.cc:5526] T 368af02998984cb2bee487b15b6a2b35 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:34.672158 21384 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 269) took 4858 ms. Trace:
I20250112 02:57:34.677707 22143 ts_tablet_manager.cc:1428] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.039s user 0.000s sys 0.004s
I20250112 02:57:34.681078 22143 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.692261 22145 tablet_bootstrap.cc:654] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.672433 21384 rpcz_store.cc:276] 0112 02:57:29.813832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.291408 (+1477576us) service_pool.cc:225] Handling call
0112 02:57:31.315677 (+ 24269us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:31.453864 (+138187us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:31.454457 (+ 593us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:31.461432 (+ 6975us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:31.498932 (+ 37500us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:34.672093 (+3173161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:31.465746 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:31.466089 (+ 343us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:31.466110 (+ 21us) write_op.cc:435] Acquired schema lock
0112 02:57:31.466124 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:31.466368 (+ 244us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:31.466397 (+ 29us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:31.466431 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:31.466571 (+ 140us) tablet.cc:698] Row locks acquired
0112 02:57:31.466585 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:31.468352 (+ 1767us) write_op.cc:270] Start()
0112 02:57:31.468444 (+ 92us) write_op.cc:276] Timestamp: P: 1736650651468251 usec, L: 0
0112 02:57:31.468461 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:31.469063 (+ 602us) log.cc:844] Serialized 405 byte log entry
0112 02:57:31.488451 (+ 19388us) op_driver.cc:464] REPLICATION: finished
0112 02:57:31.497847 (+ 9396us) write_op.cc:301] APPLY: starting
0112 02:57:31.497901 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:31.497980 (+ 79us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:31.498064 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:31.498217 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:31.498340 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:31.498380 (+ 40us) write_op.cc:312] APPLY: finished
0112 02:57:31.498552 (+ 172us) log.cc:844] Serialized 23 byte log entry
0112 02:57:31.498675 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.498731 (+ 56us) write_op.cc:454] Released schema lock
0112 02:57:31.498760 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:31.499188 (+ 428us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:31.499208 (+ 20us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":9260,"apply.run_cpu_time_us":1089,"apply.run_wall_time_us":1320,"num_ops":1,"prepare.queue_time_us":3435,"prepare.run_cpu_time_us":1886,"prepare.run_wall_time_us":8043,"replication_time_us":19907,"spinlock_wait_cycles":254080}]]}
I20250112 02:57:34.699397 22143 tablet_bootstrap.cc:654] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.725337 22143 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:34.725726 22143 ts_tablet_manager.cc:1397] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.002s
I20250112 02:57:34.731946 23155 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:34.730453 22143 raft_consensus.cc:357] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.733726 22143 raft_consensus.cc:383] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.733963 22143 raft_consensus.cc:738] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.741952 22145 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.742405 22145 ts_tablet_manager.cc:1397] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.002s
I20250112 02:57:34.749600 22145 raft_consensus.cc:357] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.750272 22145 raft_consensus.cc:383] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.750540 22145 raft_consensus.cc:738] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.756259 23155 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:34.758800 22143 consensus_queue.cc:260] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.751083 22145 consensus_queue.cc:260] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.761265 22145 ts_tablet_manager.cc:1428] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250112 02:57:34.762722 23165 tablet.cc:1721] T 21dc0755b38746a39bd773780e48cda5 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:34.765702 21389 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064] alter complete (version 1)
I20250112 02:57:34.767748 23022 consensus_queue.cc:1035] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:34.773159 23076 raft_consensus.cc:491] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.773576 23076 raft_consensus.cc:513] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.776521 22143 ts_tablet_manager.cc:1428] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.051s user 0.003s sys 0.004s
I20250112 02:57:34.777487 22143 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.777462 21388 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e] alter complete (version 2)
W20250112 02:57:34.777985 21691 consensus_peers.cc:487] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:34.778735 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:34.779867 21980 leader_election.cc:343] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.782238 23076 leader_election.cc:290] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:34.786923 23164 tablet.cc:1721] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:34.806301 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:34.807752 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:34.809008 22840 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:34.809137 21391 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a] alter complete (version 1)
I20250112 02:57:34.819548 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:34.820673 22144 tablet_bootstrap.cc:700] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Time spent opening tablet: real 0.237s user 0.001s sys 0.001s
W20250112 02:57:34.820659 21978 leader_election.cc:343] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.820969 21978 leader_election.cc:304] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.821745 23168 raft_consensus.cc:2743] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.826237 22143 tablet_bootstrap.cc:654] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.838346 22144 tablet_bootstrap.cc:654] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.838797 23155 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:34.846123 23155 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:57:34.849022 21377 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 82825c2061d74cafa814912bec005654 consensus state has no leader
I20250112 02:57:34.852301 22145 tablet_bootstrap.cc:492] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.858891 22145 tablet_bootstrap.cc:654] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:34.863823 22143 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:34.864194 22143 ts_tablet_manager.cc:1397] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.087s user 0.014s sys 0.000s
W20250112 02:57:34.864199 22840 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.190:34893, user_credentials={real_user=slave}} blocked reactor thread for 55292.4us
I20250112 02:57:34.866770 22143 raft_consensus.cc:357] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.867421 22143 raft_consensus.cc:383] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.867676 22143 raft_consensus.cc:738] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.868166 22143 consensus_queue.cc:260] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.886418 23169 raft_consensus.cc:491] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.886886 23169 raft_consensus.cc:513] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.888713 23169 leader_election.cc:290] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:34.891489 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:34.893715 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:34.894467 21980 leader_election.cc:343] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:34.896654 21978 leader_election.cc:343] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.897017 21978 leader_election.cc:304] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.898109 22143 ts_tablet_manager.cc:1428] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20250112 02:57:34.906656 23168 raft_consensus.cc:2743] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.908540 22143 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:34.916010 23022 raft_consensus.cc:491] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.916591 23022 raft_consensus.cc:513] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.919788 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:34.920989 23022 leader_election.cc:290] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.922207 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
W20250112 02:57:34.922739 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.194s user 0.002s sys 0.003s
I20250112 02:57:34.922784 22038 raft_consensus.cc:2463] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:34.923767 21690 leader_election.cc:304] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:34.924984 21691 leader_election.cc:343] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.927146 23022 raft_consensus.cc:2798] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:34.927739 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.927573 23022 raft_consensus.cc:491] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:34.928849 23022 raft_consensus.cc:3054] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.929298 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.931314 23126 consensus_queue.cc:1035] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:34.937119 23022 raft_consensus.cc:513] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.939865 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:34.941850 22144 tablet_bootstrap.cc:492] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:34.942291 22144 ts_tablet_manager.cc:1397] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.475s user 0.007s sys 0.004s
W20250112 02:57:34.947077 21691 leader_election.cc:343] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.948354 22145 tablet_bootstrap.cc:492] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:34.948804 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.948750 22145 ts_tablet_manager.cc:1397] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.013s
I20250112 02:57:34.953785 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.954371 23169 raft_consensus.cc:491] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:34.954895 23169 raft_consensus.cc:513] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.957031 22145 raft_consensus.cc:357] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.957660 22145 raft_consensus.cc:383] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:34.957917 22145 raft_consensus.cc:738] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:34.958189 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ce633c5b3b4cbea17c171564fa72d6 with delete_type TABLET_DATA_DELETED (replaced by tablet c01590521b6e4ddc9d7fcc7142d49930) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.957908 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.960996 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:34.963122 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:34.963887 22143 tablet_bootstrap.cc:654] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:34.973237 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 34) took 4747 ms (client timeout 1000 ms). Trace:
W20250112 02:57:34.973541 21390 rpcz_store.cc:269] 0112 02:57:30.225732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.608413 (+382681us) service_pool.cc:225] Handling call
0112 02:57:34.973174 (+4364761us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53013,"reported_tablets":39}
W20250112 02:57:34.973801 21691 consensus_peers.cc:487] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:34.977769 23022 leader_election.cc:290] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.962890 23169 leader_election.cc:290] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:34.963908 22014 tablet_service.cc:1514] Processing DeleteTablet for tablet 443aa544f6df4d50a78c9effd306e216 with delete_type TABLET_DATA_DELETED (replaced by tablet b9034065c6bc435d9f26f85a96297cd3) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:34.964512 21978 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.980281 23178 tablet_replica.cc:331] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:34.980307 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5e3890251e644c1eb596de043b63491e because the tablet was not found. No further retry: Not found: Tablet not found: 5e3890251e644c1eb596de043b63491e
I20250112 02:57:34.982055 23178 raft_consensus.cc:2238] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:34.982805 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4b543ed18b34cd58a1c588ed7d079c5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:34.983345 22038 raft_consensus.cc:3054] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:34.960013 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:34.976409 22996 raft_consensus.cc:491] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:34.985025 21980 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:34.984848 22996 raft_consensus.cc:513] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.984963 22145 consensus_queue.cc:260] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.985384 21980 leader_election.cc:304] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:34.986657 22996 leader_election.cc:290] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:34.987967 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:34.989585 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82825c2061d74cafa814912bec005654" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:34.990675 23168 raft_consensus.cc:2743] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:34.992795 22145 ts_tablet_manager.cc:1428] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20250112 02:57:34.993168 22144 raft_consensus.cc:357] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:34.993674 22145 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:34.993952 22144 raft_consensus.cc:383] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.001259 23179 tablet.cc:1721] T 21dc0755b38746a39bd773780e48cda5 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:35.002013 22038 raft_consensus.cc:2463] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:35.009721 22143 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:35.010108 22143 ts_tablet_manager.cc:1397] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.003s
I20250112 02:57:35.012286 22143 raft_consensus.cc:357] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.012897 22143 raft_consensus.cc:383] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.013144 22143 raft_consensus.cc:738] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.014381 21384 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 21dc0755b38746a39bd773780e48cda5 (table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682]). Expected version 1 got 0
I20250112 02:57:35.016295 21384 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:35.017503 21384 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:35.018707 23109 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:35.018967 23109 catalog_manager.cc:1553] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: loading cluster ID for follower catalog manager: success
I20250112 02:57:35.019181 21690 leader_election.cc:304] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.019847 23022 raft_consensus.cc:2798] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:35.020251 23022 raft_consensus.cc:695] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:35.020989 23022 consensus_queue.cc:237] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.030594 22143 consensus_queue.cc:260] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.032243 22145 tablet_bootstrap.cc:654] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.036495 22143 ts_tablet_manager.cc:1428] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.026s user 0.003s sys 0.004s
I20250112 02:57:35.037267 22143 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:35.039165 21392 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9] alter complete (version 2)
W20250112 02:57:35.054124 21388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 40) took 3034 ms (client timeout 1000 ms). Trace:
I20250112 02:57:35.060074 22145 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:35.060453 22145 ts_tablet_manager.cc:1397] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.003s
I20250112 02:57:35.062611 22144 raft_consensus.cc:738] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.062852 23168 raft_consensus.cc:491] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.063283 23168 raft_consensus.cc:513] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.063320 22144 consensus_queue.cc:260] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.065645 23168 leader_election.cc:290] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:34.994966 21835 leader_election.cc:304] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:35.066545 22144 ts_tablet_manager.cc:1428] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.076s user 0.003s sys 0.001s
I20250112 02:57:35.066828 23178 raft_consensus.cc:2267] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:35.067004 21835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 72165.6us
I20250112 02:57:35.043756 21901 raft_consensus.cc:3054] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:35.070647 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250112 02:57:35.070825 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247f0ef0ada44d9d957c8ff42b0da4b5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:35.069077 22145 raft_consensus.cc:357] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.075289 22145 raft_consensus.cc:383] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.075542 22145 raft_consensus.cc:738] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.069816 23178 ts_tablet_manager.cc:1905] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:57:35.072216 21980 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.079088 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.079383 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.082118 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247f0ef0ada44d9d957c8ff42b0da4b5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:35.083418 21978 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.083525 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.083809 21978 leader_election.cc:304] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.084770 23168 raft_consensus.cc:2743] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.082726 22145 consensus_queue.cc:260] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.087201 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.087869 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 7de9ed3d1552490da7ab76037d7debc0 with delete_type TABLET_DATA_DELETED (replaced by tablet b4aff0623f504bf09011ca6de28a3b82) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.089061 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.090385 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.090440 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.097484 22145 ts_tablet_manager.cc:1428] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.037s user 0.006s sys 0.001s
W20250112 02:57:35.077752 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 41) took 2011 ms (client timeout 1000 ms). Trace:
W20250112 02:57:35.097781 21383 rpcz_store.cc:269] 0112 02:57:33.066078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:33.185563 (+119485us) service_pool.cc:225] Handling call
0112 02:57:35.077685 (+1892122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28838,"reported_tablets":26}
I20250112 02:57:35.100638 23168 raft_consensus.cc:491] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.101198 23168 raft_consensus.cc:513] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.102006 22145 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:35.105428 23168 leader_election.cc:290] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:35.105988 22143 tablet_bootstrap.cc:654] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.105789 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:35.107998 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
W20250112 02:57:35.108220 21978 leader_election.cc:343] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:35.113327 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 42) took 3497 ms (client timeout 1000 ms). Trace:
I20250112 02:57:35.114019 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.114711 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:35.115317 23186 tablet_replica.cc:331] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
W20250112 02:57:35.116322 21980 leader_election.cc:343] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.116681 21980 leader_election.cc:304] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:35.094738 21388 rpcz_store.cc:269] 0112 02:57:32.020043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.121354 (+101311us) service_pool.cc:225] Handling call
0112 02:57:32.132507 (+ 11153us) spinlock_profiling.cc:243] Waited 7.44 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:34.776607 (+2644100us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.776632 (+ 25us) write_op.cc:454] Released schema lock
0112 02:57:35.054059 (+277427us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:32.587347 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:32.587768 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:32.587792 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:32.587807 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:32.588098 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:32.588139 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:32.588165 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:32.588366 (+ 201us) tablet.cc:698] Row locks acquired
0112 02:57:32.588383 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:32.588493 (+ 110us) write_op.cc:270] Start()
0112 02:57:32.588590 (+ 97us) write_op.cc:276] Timestamp: P: 1736650652588581 usec, L: 0
0112 02:57:32.588610 (+ 20us) op_driver.cc:348] REPLICATION: starting
0112 02:57:32.589191 (+ 581us) log.cc:844] Serialized 536 byte log entry
0112 02:57:32.598532 (+ 9341us) op_driver.cc:464] REPLICATION: finished
0112 02:57:32.598924 (+ 392us) write_op.cc:301] APPLY: starting
0112 02:57:32.598975 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:32.599076 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:32.599169 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:32.599352 (+ 183us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:32.599475 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:32.599511 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:32.599666 (+ 155us) log.cc:844] Serialized 23 byte log entry
0112 02:57:32.599790 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.599851 (+ 61us) write_op.cc:454] Released schema lock
0112 02:57:32.599877 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:32.600086 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.600107 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:34.735840 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.736254 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.736297 (+ 43us) write_op.cc:435] Acquired schema lock
0112 02:57:34.736314 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:34.736608 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.736646 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.736671 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:34.736828 (+ 157us) tablet.cc:698] Row locks acquired
0112 02:57:34.736839 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:34.736964 (+ 125us) write_op.cc:270] Start()
0112 02:57:34.737061 (+ 97us) write_op.cc:276] Timestamp: P: 1736650654736927 usec, L: 0
0112 02:57:34.737079 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.737566 (+ 487us) log.cc:844] Serialized 356 byte log entry
0112 02:57:34.757976 (+ 20410us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.758611 (+ 635us) write_op.cc:301] APPLY: starting
0112 02:57:34.758661 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.758744 (+ 83us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.758834 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.758997 (+ 163us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.759113 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:34.759148 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:57:34.759281 (+ 133us) log.cc:844] Serialized 23 byte log entry
0112 02:57:34.759385 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.759429 (+ 44us) write_op.cc:454] Released schema lock
0112 02:57:34.759452 (+ 23us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":22,"spinlock_wait_cycles":17116032,"child_traces":[["op",{"apply.queue_time_us":258,"apply.run_cpu_time_us":1120,"apply.run_wall_time_us":1120,"num_ops":1,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":2129,"prepare.run_wall_time_us":3120,"raft.queue_time_us":1383,"raft.run_cpu_time_us":631,"raft.run_wall_time_us":13688,"replication_time_us":9811,"spinlock_wait_cycles":67456}],["op",{"apply.queue_time_us":467,"apply.run_cpu_time_us":990,"apply.run_wall_time_us":989,"num_ops":1,"prepare.queue_time_us":19181,"prepare.run_cpu_time_us":1922,"prepare.run_wall_time_us":1921,"replication_time_us":20811}]]}
I20250112 02:57:35.117568 23168 raft_consensus.cc:2743] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.125860 22143 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:35.126340 22143 ts_tablet_manager.cc:1397] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.089s user 0.011s sys 0.000s
I20250112 02:57:35.128382 21388 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_ecb7cc54618b492cbaf125b216cf5ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:35.128916 22143 raft_consensus.cc:357] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.129626 22143 raft_consensus.cc:383] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.129915 22143 raft_consensus.cc:738] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.137430 23186 raft_consensus.cc:2238] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:35.137722 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9543a0a0359e4b72bd30f1cb6290f767 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.139719 23186 raft_consensus.cc:2267] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:35.143654 23186 ts_tablet_manager.cc:1905] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.143954 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa732d27f144aaad088ecc072837e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.145511 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet c48d1bc147ba48ebbebfd707dc44fe02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:20 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.148859 22144 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.150631 22996 raft_consensus.cc:2743] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.152307 23184 tablet_replica.cc:331] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:35.153203 23184 raft_consensus.cc:2238] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:35.153635 23184 raft_consensus.cc:2267] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:35.148485 23022 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.170571 23022 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.173017 23022 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.155002 21730 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.155917 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.156646 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:35.175601 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.177235 22038 raft_consensus.cc:2463] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.178177 21690 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:35.178952 23022 raft_consensus.cc:2798] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.179260 23022 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.179560 23022 raft_consensus.cc:3054] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.186820 22143 consensus_queue.cc:260] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.189095 22143 ts_tablet_manager.cc:1428] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.062s user 0.004s sys 0.000s
I20250112 02:57:35.192299 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.199505 21691 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.208765 23178 ts_tablet_manager.cc:1918] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:35.209232 23178 log.cc:1198] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:35.210042 23178 ts_tablet_manager.cc:1939] T 2601daad8ba742ec88ccfe603440e8e7 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:35.212424 22144 tablet_bootstrap.cc:654] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.214648 22145 tablet_bootstrap.cc:654] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.215679 22008 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:35.216275 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:35.218132 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c16031f5b74cf9885fac110f445e1a with delete_type TABLET_DATA_DELETED (replaced by tablet da54d9dbab2f416785ec0379095bcf3e) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:35.143359 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 39) took 4642 ms (client timeout 1000 ms). Trace:
W20250112 02:57:35.223573 21388 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ecb7cc54618b492cbaf125b216cf5ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:35.219645 21389 rpcz_store.cc:269] 0112 02:57:30.500707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.648308 (+147601us) service_pool.cc:225] Handling call
0112 02:57:35.143283 (+4494975us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:34.702786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.703207 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.703232 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:34.703245 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:34.703520 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.703565 (+ 45us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.703587 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:34.703736 (+ 149us) tablet.cc:698] Row locks acquired
0112 02:57:34.703748 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:34.703860 (+ 112us) write_op.cc:270] Start()
0112 02:57:34.703949 (+ 89us) write_op.cc:276] Timestamp: P: 1736650654703824 usec, L: 0
0112 02:57:34.703961 (+ 12us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.704457 (+ 496us) log.cc:844] Serialized 356 byte log entry
0112 02:57:34.746546 (+ 42089us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.746859 (+ 313us) write_op.cc:301] APPLY: starting
0112 02:57:34.746913 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.747004 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.747095 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.747261 (+ 166us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.747377 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:34.747421 (+ 44us) write_op.cc:312] APPLY: finished
0112 02:57:34.747585 (+ 164us) log.cc:844] Serialized 23 byte log entry
0112 02:57:34.747705 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.747754 (+ 49us) write_op.cc:454] Released schema lock
0112 02:57:34.747778 (+ 24us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:34.747969 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.747988 (+ 19us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":50911,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":180,"apply.run_cpu_time_us":1080,"apply.run_wall_time_us":1080,"num_ops":1,"prepare.queue_time_us":91,"prepare.run_cpu_time_us":1959,"prepare.run_wall_time_us":1958,"raft.queue_time_us":181,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":587,"replication_time_us":42480}]]}
W20250112 02:57:35.113691 21391 rpcz_store.cc:269] 0112 02:57:31.615783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.623820 (+ 8037us) service_pool.cc:225] Handling call
0112 02:57:35.113264 (+3489444us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:32.666622 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:32.667609 (+ 987us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:32.667637 (+ 28us) write_op.cc:435] Acquired schema lock
0112 02:57:32.667654 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:32.668130 (+ 476us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:32.668174 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:32.668200 (+ 26us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:32.668488 (+ 288us) tablet.cc:698] Row locks acquired
0112 02:57:32.668501 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:32.668612 (+ 111us) write_op.cc:270] Start()
0112 02:57:32.668807 (+ 195us) write_op.cc:276] Timestamp: P: 1736650652668578 usec, L: 0
0112 02:57:32.668822 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:32.669337 (+ 515us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:32.677046 (+ 7709us) op_driver.cc:464] REPLICATION: finished
0112 02:57:32.677453 (+ 407us) write_op.cc:301] APPLY: starting
0112 02:57:32.677507 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:32.677627 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:32.677716 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:32.678028 (+ 312us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:32.678157 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:32.678198 (+ 41us) write_op.cc:312] APPLY: finished
0112 02:57:32.678376 (+ 178us) log.cc:844] Serialized 35 byte log entry
0112 02:57:32.679806 (+ 1430us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.679899 (+ 93us) write_op.cc:454] Released schema lock
0112 02:57:32.679939 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:32.682944 (+ 3005us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.682968 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:34.738823 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.739214 (+ 391us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.739237 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:34.739254 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:34.739543 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.739584 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.739608 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:34.739770 (+ 162us) tablet.cc:698] Row locks acquired
0112 02:57:34.739783 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:34.739890 (+ 107us) write_op.cc:270] Start()
0112 02:57:34.739972 (+ 82us) write_op.cc:276] Timestamp: P: 1736650654739869 usec, L: 0
0112 02:57:34.739990 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.740441 (+ 451us) log.cc:844] Serialized 356 byte log entry
0112 02:57:34.758182 (+ 17741us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.759682 (+ 1500us) write_op.cc:301] APPLY: starting
0112 02:57:34.759722 (+ 40us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.759791 (+ 69us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.759874 (+ 83us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.760012 (+ 138us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.760129 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:34.760165 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:34.760293 (+ 128us) log.cc:844] Serialized 23 byte log entry
0112 02:57:34.760390 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.760439 (+ 49us) write_op.cc:454] Released schema lock
0112 02:57:34.760462 (+ 23us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":80646,"reported_tablets":35,"child_traces":[["op",{"apply.queue_time_us":242,"apply.run_cpu_time_us":1426,"apply.run_wall_time_us":2683,"num_ops":3,"prepare.queue_time_us":138,"prepare.run_cpu_time_us":2958,"prepare.run_wall_time_us":16314,"raft.queue_time_us":101,"raft.run_cpu_time_us":610,"raft.run_wall_time_us":610,"replication_time_us":8105}],["op",{"apply.queue_time_us":1396,"apply.run_cpu_time_us":919,"apply.run_wall_time_us":919,"mutex_wait_us":19767,"num_ops":1,"prepare.queue_time_us":19892,"prepare.run_cpu_time_us":1778,"prepare.run_wall_time_us":2544,"replication_time_us":18129,"thread_start_us":89983,"threads_started":1}]]}
W20250112 02:57:35.199520 21385 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 44) took 1564 ms (client timeout 1000 ms). Trace:
W20250112 02:57:35.231782 21385 rpcz_store.cc:269] 0112 02:57:33.635168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:34.553586 (+918418us) service_pool.cc:225] Handling call
0112 02:57:35.199457 (+645871us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:34.854747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.855092 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.855115 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:34.855133 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:34.855704 (+ 571us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.855742 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.855763 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:34.856032 (+ 269us) tablet.cc:698] Row locks acquired
0112 02:57:34.856050 (+ 18us) write_op.cc:260] PREPARE: finished
0112 02:57:34.856156 (+ 106us) write_op.cc:270] Start()
0112 02:57:34.856253 (+ 97us) write_op.cc:276] Timestamp: P: 1736650654856181 usec, L: 0
0112 02:57:34.856270 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.856739 (+ 469us) log.cc:844] Serialized 919 byte log entry
0112 02:57:34.900625 (+ 43886us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.902120 (+ 1495us) write_op.cc:301] APPLY: starting
0112 02:57:34.902161 (+ 41us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.902246 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.902322 (+ 76us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.902493 (+ 171us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.902622 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:34.902657 (+ 35us) write_op.cc:312] APPLY: finished
0112 02:57:34.902788 (+ 131us) log.cc:844] Serialized 29 byte log entry
0112 02:57:34.902886 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.902937 (+ 51us) write_op.cc:454] Released schema lock
0112 02:57:34.902964 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:34.903131 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.903152 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41932,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":1397,"apply.run_cpu_time_us":990,"apply.run_wall_time_us":989,"num_ops":2,"prepare.queue_time_us":36182,"prepare.run_cpu_time_us":2177,"prepare.run_wall_time_us":3117,"replication_time_us":44286}]]}
W20250112 02:57:35.206826 21387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 43) took 2894 ms (client timeout 1000 ms). Trace:
I20250112 02:57:35.234861 23022 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:35.234920 21387 rpcz_store.cc:269] 0112 02:57:32.312382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.312505 (+ 123us) service_pool.cc:225] Handling call
0112 02:57:35.206762 (+2894257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:32.824128 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:32.824521 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:32.824547 (+ 26us) write_op.cc:435] Acquired schema lock
0112 02:57:32.824566 (+ 19us) tablet.cc:659] Decoding operations
0112 02:57:32.825062 (+ 496us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:32.825106 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:32.825129 (+ 23us) tablet.cc:682] Acquiring locks for 4 operations
0112 02:57:32.825442 (+ 313us) tablet.cc:698] Row locks acquired
0112 02:57:32.825458 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:32.825569 (+ 111us) write_op.cc:270] Start()
0112 02:57:32.825650 (+ 81us) write_op.cc:276] Timestamp: P: 1736650652825597 usec, L: 0
0112 02:57:32.825666 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:57:32.826166 (+ 500us) log.cc:844] Serialized 1686 byte log entry
0112 02:57:32.833951 (+ 7785us) op_driver.cc:464] REPLICATION: finished
0112 02:57:32.834717 (+ 766us) write_op.cc:301] APPLY: starting
0112 02:57:32.834768 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:32.835168 (+ 400us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:32.835268 (+ 100us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:32.835630 (+ 362us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:32.835760 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0112 02:57:32.835796 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:32.835998 (+ 202us) log.cc:844] Serialized 41 byte log entry
0112 02:57:32.836130 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.836197 (+ 67us) write_op.cc:454] Released schema lock
0112 02:57:32.836223 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:32.836422 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:32.836444 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":143918,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":634,"apply.run_cpu_time_us":1661,"apply.run_wall_time_us":1660,"num_ops":4,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":2315,"prepare.run_wall_time_us":2313,"raft.queue_time_us":1326,"raft.run_cpu_time_us":532,"raft.run_wall_time_us":532,"replication_time_us":8202}]]}
I20250112 02:57:35.238950 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:35.240110 21691 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.240758 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_d0fc06e00a274dc99d145cadb6530068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:35.255596 21901 raft_consensus.cc:3054] T 118ef6d7c16941598177b92a94c4b472 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.268615 22143 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:35.281363 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 2601daad8ba742ec88ccfe603440e8e7 (table default.Table_33ea8d4190104eb8837824b516dd2314 [id=097357eb9ce548aa9fe2db1a906c70d8]) successfully deleted
I20250112 02:57:35.281389 23184 ts_tablet_manager.cc:1905] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.282610 23022 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.290005 21385 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_45510e3d77df4fa899940125a4a5da9c" } modify_external_catalogs: true
I20250112 02:57:35.290481 21385 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_45510e3d77df4fa899940125a4a5da9c" } modify_external_catalogs: true
W20250112 02:57:35.291393 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because the tablet was not found. No further retry: Not found: Tablet not found: 84f7700b8cce4261a32f2009d1481b99
W20250112 02:57:35.292055 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because the tablet was not found. No further retry: Not found: Tablet not found: 96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:35.292601 22145 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:35.292639 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet a2ce633c5b3b4cbea17c171564fa72d6 because the tablet was not found. No further retry: Not found: Tablet not found: a2ce633c5b3b4cbea17c171564fa72d6
I20250112 02:57:35.293090 22145 ts_tablet_manager.cc:1397] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.191s user 0.011s sys 0.000s
W20250112 02:57:35.293229 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 27f0d6a5afd140a8af77a5a39dc39dd2 because the tablet was not found. No further retry: Not found: Tablet not found: 27f0d6a5afd140a8af77a5a39dc39dd2
I20250112 02:57:35.293419 23178 tablet_replica.cc:331] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:35.294355 23178 raft_consensus.cc:2238] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:35.297560 22145 raft_consensus.cc:357] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:35.242343 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0fc06e00a274dc99d145cadb6530068 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:57:35.219254 21392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 35) took 3832 ms (client timeout 1000 ms). Trace:
W20250112 02:57:35.310937 21392 rpcz_store.cc:269] 0112 02:57:31.386588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:31.581674 (+195086us) service_pool.cc:225] Handling call
0112 02:57:35.219186 (+3637512us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:34.773992 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.774491 (+ 499us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.774529 (+ 38us) write_op.cc:435] Acquired schema lock
0112 02:57:34.774544 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:34.774797 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.774838 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.774860 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:34.775007 (+ 147us) tablet.cc:698] Row locks acquired
0112 02:57:34.775018 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:34.775131 (+ 113us) write_op.cc:270] Start()
0112 02:57:34.775208 (+ 77us) write_op.cc:276] Timestamp: P: 1736650654775125 usec, L: 0
0112 02:57:34.775223 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.852668 (+ 77445us) log.cc:844] Serialized 356 byte log entry
0112 02:57:34.854534 (+ 1866us) spinlock_profiling.cc:243] Waited 76.9 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b66d1 00007f7eed62d198 00007f7eeda993b1 00007f7eeda98b88 00007f7eeda9d821 00007f7eeda9d7d9 00007f7eeda9d769 00007f7eeda9d731 00007f7eeda9c95d 000000000059ea04 000000000059e839
0112 02:57:34.900406 (+ 45872us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.900940 (+ 534us) write_op.cc:301] APPLY: starting
0112 02:57:34.901001 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.901083 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.901100 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.901247 (+ 147us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.901355 (+ 108us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:34.901389 (+ 34us) write_op.cc:312] APPLY: finished
0112 02:57:34.901525 (+ 136us) log.cc:844] Serialized 23 byte log entry
0112 02:57:34.901640 (+ 115us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.901693 (+ 53us) write_op.cc:454] Released schema lock
0112 02:57:34.901720 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:34.901908 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.901930 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24710,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":383,"apply.run_cpu_time_us":946,"apply.run_wall_time_us":963,"mutex_wait_us":17819,"num_ops":1,"prepare.queue_time_us":48997,"prepare.run_cpu_time_us":3442,"prepare.run_wall_time_us":80654,"replication_time_us":125099,"spinlock_wait_cycles":176882432}]]}
I20250112 02:57:35.315114 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_987a7a8f641b4473bc75c11ec9f5fff9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:35.316057 21691 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 43744.5us
I20250112 02:57:35.318176 21391 master_service.cc:946] ReplaceTablet: received request to replace tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:35.318615 22145 raft_consensus.cc:383] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.318897 22145 raft_consensus.cc:738] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.319470 22145 consensus_queue.cc:260] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.294952 23178 raft_consensus.cc:2267] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:35.323738 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:35.324422 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 36) took 2824 ms (client timeout 1000 ms). Trace:
W20250112 02:57:35.331454 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 53717.9us
I20250112 02:57:35.333262 22145 ts_tablet_manager.cc:1428] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250112 02:57:35.333945 23165 tablet.cc:1721] T 82825c2061d74cafa814912bec005654 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:35.335853 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:35.336859 23178 ts_tablet_manager.cc:1905] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.337831 23179 tablet.cc:1721] T 82825c2061d74cafa814912bec005654 P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:35.342715 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.085s user 0.003s sys 0.000s
I20250112 02:57:35.345113 23186 ts_tablet_manager.cc:1918] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:35.345459 23186 log.cc:1198] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:35.346210 23186 ts_tablet_manager.cc:1939] T 96f98f6840e24f3c9d03937c1bbc00e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:35.348062 22143 tablet_bootstrap.cc:654] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:35.349207 21690 consensus_peers.cc:487] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2601daad8ba742ec88ccfe603440e8e7. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:35.349838 21690 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.350261 21690 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:35.352676 23164 tablet.cc:1721] T 82825c2061d74cafa814912bec005654 P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:35.355429 23192 raft_consensus.cc:2743] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.355633 22144 tablet_bootstrap.cc:492] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:35.355995 22144 ts_tablet_manager.cc:1397] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.207s user 0.009s sys 0.000s
I20250112 02:57:35.356173 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 96f98f6840e24f3c9d03937c1bbc00e7 (table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]) successfully deleted
W20250112 02:57:35.356897 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 7de9ed3d1552490da7ab76037d7debc0 because the tablet was not found. No further retry: Not found: Tablet not found: 7de9ed3d1552490da7ab76037d7debc0
I20250112 02:57:35.358269 22144 raft_consensus.cc:357] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.358950 22144 raft_consensus.cc:383] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.371272 23192 raft_consensus.cc:491] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.371726 23192 raft_consensus.cc:513] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.373593 23192 leader_election.cc:290] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.374032 23192 raft_consensus.cc:491] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.375514 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.376920 21691 leader_election.cc:343] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.394066 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.394904 22044 raft_consensus.cc:2463] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.394150 22144 raft_consensus.cc:738] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
W20250112 02:57:35.324716 21386 rpcz_store.cc:269] 0112 02:57:32.500297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:32.520470 (+ 20173us) service_pool.cc:225] Handling call
0112 02:57:35.324367 (+2803897us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:34.550247 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:34.550650 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:34.550675 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:34.550701 (+ 26us) tablet.cc:659] Decoding operations
0112 02:57:34.550971 (+ 270us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:34.551013 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:34.551036 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:34.551187 (+ 151us) tablet.cc:698] Row locks acquired
0112 02:57:34.551199 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:34.551314 (+ 115us) write_op.cc:270] Start()
0112 02:57:34.551395 (+ 81us) write_op.cc:276] Timestamp: P: 1736650654551422 usec, L: 0
0112 02:57:34.551408 (+ 13us) op_driver.cc:348] REPLICATION: starting
0112 02:57:34.551898 (+ 490us) log.cc:844] Serialized 536 byte log entry
0112 02:57:34.560399 (+ 8501us) op_driver.cc:464] REPLICATION: finished
0112 02:57:34.560678 (+ 279us) write_op.cc:301] APPLY: starting
0112 02:57:34.560729 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:34.560815 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:34.560909 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:34.561049 (+ 140us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:34.561162 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:34.561198 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:34.561356 (+ 158us) log.cc:844] Serialized 23 byte log entry
0112 02:57:34.561487 (+ 131us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.561541 (+ 54us) write_op.cc:454] Released schema lock
0112 02:57:34.561567 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:34.561750 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:34.561771 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":255094,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":157,"apply.run_cpu_time_us":1035,"apply.run_wall_time_us":1034,"num_ops":1,"prepare.queue_time_us":1176,"prepare.run_cpu_time_us":1915,"prepare.run_wall_time_us":1914,"raft.queue_time_us":913,"raft.run_cpu_time_us":588,"raft.run_wall_time_us":588,"replication_time_us":8905}]]}
I20250112 02:57:35.402383 21386 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ff93d70a357e442dafed35cd75ae6152" } modify_external_catalogs: true
W20250112 02:57:35.405238 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_987a7a8f641b4473bc75c11ec9f5fff9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:35.408154 22144 consensus_queue.cc:260] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.411232 23197 raft_consensus.cc:491] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.411652 23197 raft_consensus.cc:513] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.413260 23197 leader_election.cc:290] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.420061 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.421234 21691 leader_election.cc:343] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:35.421339 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.076s user 0.002s sys 0.000s
I20250112 02:57:35.374457 23192 raft_consensus.cc:513] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.423863 23192 leader_election.cc:290] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.425192 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.426376 21691 leader_election.cc:343] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.437736 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.438302 22038 raft_consensus.cc:2463] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.440416 23197 raft_consensus.cc:491] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.440886 23197 raft_consensus.cc:513] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.444010 21690 leader_election.cc:304] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.446050 23124 raft_consensus.cc:2798] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.446368 23124 raft_consensus.cc:491] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.446698 23124 raft_consensus.cc:3054] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.448792 22144 ts_tablet_manager.cc:1428] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.093s user 0.008s sys 0.000s
I20250112 02:57:35.449618 22144 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.453086 23124 raft_consensus.cc:513] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.454813 23124 leader_election.cc:290] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.455981 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:35.459026 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:35.461048 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.461594 22044 raft_consensus.cc:2463] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.468473 22145 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:35.473251 22145 tablet_bootstrap.cc:654] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.355455 23186 tablet_replica.cc:331] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:35.476430 23184 ts_tablet_manager.cc:1918] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:35.476847 23184 log.cc:1198] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/5e3890251e644c1eb596de043b63491e
I20250112 02:57:35.477277 21388 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 288) took 2986 ms. Trace:
I20250112 02:57:35.477490 23184 ts_tablet_manager.cc:1939] T 5e3890251e644c1eb596de043b63491e P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:35.477599 21388 rpcz_store.cc:276] 0112 02:57:32.490896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.127824 (+2636928us) service_pool.cc:225] Handling call
0112 02:57:35.270709 (+142885us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:35.272154 (+ 1445us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:35.317271 (+ 45117us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.317307 (+ 36us) write_op.cc:454] Released schema lock
0112 02:57:35.317491 (+ 184us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:35.454677 (+137186us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:35.477200 (+ 22523us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.273235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.273590 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.273613 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:35.273628 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:35.273893 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.273929 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.273954 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:35.274134 (+ 180us) tablet.cc:698] Row locks acquired
0112 02:57:35.274148 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:35.274248 (+ 100us) write_op.cc:270] Start()
0112 02:57:35.274324 (+ 76us) write_op.cc:276] Timestamp: P: 1736650655274237 usec, L: 0
0112 02:57:35.274343 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.274861 (+ 518us) log.cc:844] Serialized 433 byte log entry
0112 02:57:35.295359 (+ 20498us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.296395 (+ 1036us) write_op.cc:301] APPLY: starting
0112 02:57:35.296458 (+ 63us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.296570 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.296682 (+ 112us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.297174 (+ 492us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.297342 (+ 168us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:35.297391 (+ 49us) write_op.cc:312] APPLY: finished
0112 02:57:35.297572 (+ 181us) log.cc:844] Serialized 29 byte log entry
0112 02:57:35.297705 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.297778 (+ 73us) write_op.cc:454] Released schema lock
0112 02:57:35.297821 (+ 43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":862,"apply.run_cpu_time_us":1623,"apply.run_wall_time_us":1621,"num_ops":2,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1882,"prepare.run_wall_time_us":1881,"raft.queue_time_us":591,"raft.run_cpu_time_us":636,"raft.run_wall_time_us":636,"replication_time_us":20922}]]}
I20250112 02:57:35.480868 21388 master_service.cc:946] ReplaceTablet: received request to replace tablet 03d2021384bb41b58a66fcb7afaeaca8 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:35.482208 21391 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50642 (request call id 292) took 2326 ms. Trace:
I20250112 02:57:35.482556 21391 rpcz_store.cc:276] 0112 02:57:33.156260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.318060 (+2161800us) service_pool.cc:225] Handling call
0112 02:57:35.482142 (+164082us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:35.483886 23186 raft_consensus.cc:2238] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:35.483631 21391 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_e0781749d01044d5afbd6e4bb09d1c9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:57:35.485436 23186 raft_consensus.cc:2267] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:35.485531 21391 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0781749d01044d5afbd6e4bb09d1c9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:35.491712 22144 tablet_bootstrap.cc:654] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:35.462452 21691 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:35.494966 21691 leader_election.cc:343] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.471073 21690 leader_election.cc:304] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.497146 22145 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:35.497516 22145 ts_tablet_manager.cc:1397] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.006s
I20250112 02:57:35.499468 22145 raft_consensus.cc:357] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.500038 22145 raft_consensus.cc:383] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.500245 22145 raft_consensus.cc:738] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.501650 22145 consensus_queue.cc:260] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.502480 22144 tablet_bootstrap.cc:492] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:35.502868 22144 ts_tablet_manager.cc:1397] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.001s
I20250112 02:57:35.503401 22145 ts_tablet_manager.cc:1428] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250112 02:57:35.504005 23196 raft_consensus.cc:2798] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.504212 22145 tablet_bootstrap.cc:492] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:35.504405 23196 raft_consensus.cc:491] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.504788 23196 raft_consensus.cc:3054] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.504895 22144 raft_consensus.cc:357] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.505535 22144 raft_consensus.cc:383] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.505594 23178 ts_tablet_manager.cc:1918] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:35.505793 22144 raft_consensus.cc:738] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.505944 23178 log.cc:1198] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/443aa544f6df4d50a78c9effd306e216
I20250112 02:57:35.506409 22144 consensus_queue.cc:260] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.509406 22144 ts_tablet_manager.cc:1428] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250112 02:57:35.510221 22144 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.513980 22145 tablet_bootstrap.cc:654] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.457330 23197 leader_election.cc:290] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.521909 22144 tablet_bootstrap.cc:654] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.529050 22145 tablet_bootstrap.cc:492] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:35.529418 22145 ts_tablet_manager.cc:1397] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20250112 02:57:35.531646 21690 leader_election.cc:304] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.531510 22145 raft_consensus.cc:357] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.532106 22145 raft_consensus.cc:383] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.532330 22145 raft_consensus.cc:738] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.533028 22145 consensus_queue.cc:260] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.542937 22145 ts_tablet_manager.cc:1428] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250112 02:57:35.543751 22145 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:35.546900 23124 raft_consensus.cc:2798] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.547590 23124 raft_consensus.cc:491] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.548866 22145 tablet_bootstrap.cc:654] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.549221 23124 raft_consensus.cc:3054] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.550571 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:35.551131 22044 raft_consensus.cc:3054] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.552840 22144 tablet_bootstrap.cc:492] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:35.553241 22144 ts_tablet_manager.cc:1397] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.011s
I20250112 02:57:35.554203 21392 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 291) took 2446 ms. Trace:
I20250112 02:57:35.555378 22144 raft_consensus.cc:357] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:35.536278 21691 consensus_peers.cc:487] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:35.555982 22144 raft_consensus.cc:383] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.556229 22144 raft_consensus.cc:738] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.556774 22144 consensus_queue.cc:260] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.558138 21899 raft_consensus.cc:3054] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.559751 21387 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 289) took 2836 ms. Trace:
I20250112 02:57:35.563475 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.564579 22038 raft_consensus.cc:2463] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.494402 23186 ts_tablet_manager.cc:1905] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.554538 21392 rpcz_store.cc:276] 0112 02:57:33.107923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.314705 (+2206782us) service_pool.cc:225] Handling call
0112 02:57:35.522808 (+208103us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:35.524487 (+ 1679us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:35.553393 (+ 28906us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.553419 (+ 26us) write_op.cc:454] Released schema lock
0112 02:57:35.553618 (+ 199us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:35.553913 (+ 295us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:35.554148 (+ 235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.525949 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.526459 (+ 510us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.526494 (+ 35us) write_op.cc:435] Acquired schema lock
0112 02:57:35.526505 (+ 11us) tablet.cc:659] Decoding operations
0112 02:57:35.526868 (+ 363us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.526919 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.526944 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:35.527757 (+ 813us) tablet.cc:698] Row locks acquired
0112 02:57:35.527771 (+ 14us) write_op.cc:260] PREPARE: finished
0112 02:57:35.527898 (+ 127us) write_op.cc:270] Start()
0112 02:57:35.527984 (+ 86us) write_op.cc:276] Timestamp: P: 1736650655527984 usec, L: 0
0112 02:57:35.528003 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.528529 (+ 526us) log.cc:844] Serialized 433 byte log entry
0112 02:57:35.538226 (+ 9697us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.538725 (+ 499us) write_op.cc:301] APPLY: starting
0112 02:57:35.538795 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.538923 (+ 128us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.539050 (+ 127us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.539498 (+ 448us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.539685 (+ 187us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:35.539743 (+ 58us) write_op.cc:312] APPLY: finished
0112 02:57:35.539943 (+ 200us) log.cc:844] Serialized 29 byte log entry
0112 02:57:35.540114 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.540198 (+ 84us) write_op.cc:454] Released schema lock
0112 02:57:35.540234 (+ 36us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":1742,"apply.run_wall_time_us":1741,"num_ops":2,"prepare.queue_time_us":112,"prepare.run_cpu_time_us":2892,"prepare.run_wall_time_us":4298,"raft.queue_time_us":89,"raft.run_cpu_time_us":582,"raft.run_wall_time_us":582,"replication_time_us":10121}]]}
I20250112 02:57:35.566799 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_706a3ed7905244fbbf03303bdbde0fb9" } new_table_name: "default.Table_54f73264193a43cba9cb43f55fa18e09" modify_external_catalogs: true
I20250112 02:57:35.567765 22144 ts_tablet_manager.cc:1428] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20250112 02:57:35.568604 22144 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.570662 21386 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_ff93d70a357e442dafed35cd75ae6152" } modify_external_catalogs: true
W20250112 02:57:35.571861 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20250112 02:57:35.574218 22144 tablet_bootstrap.cc:654] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.576655 22143 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:35.577037 22143 ts_tablet_manager.cc:1397] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.309s user 0.004s sys 0.006s
W20250112 02:57:35.581977 21384 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 42) took 1488 ms (client timeout 1000 ms). Trace:
I20250112 02:57:35.582347 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:35.583184 22143 raft_consensus.cc:357] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.583848 22143 raft_consensus.cc:383] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.584113 22143 raft_consensus.cc:738] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.584640 22143 consensus_queue.cc:260] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.592164 21690 leader_election.cc:304] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:35.592586 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eab717cb5d794d15b9b51881b86501fc consensus state has no leader
I20250112 02:57:35.592774 23022 raft_consensus.cc:2798] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.593061 23022 raft_consensus.cc:491] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.593333 23022 raft_consensus.cc:3054] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.594229 21392 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_0d55eaca1ca1484a8b8840ad492e7529" } new_table_name: "default.Table_03e04d34d58e463e93d16e858752d3cb" modify_external_catalogs: true
I20250112 02:57:35.637917 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 5e3890251e644c1eb596de043b63491e (table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]) successfully deleted
W20250112 02:57:35.639752 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 15c16031f5b74cf9885fac110f445e1a because the tablet was not found. No further retry: Not found: Tablet not found: 15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:35.640720 23184 tablet_replica.cc:331] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:35.641638 23184 raft_consensus.cc:2238] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:35.642153 23184 raft_consensus.cc:2267] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:35.644019 23184 ts_tablet_manager.cc:1905] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.609715 23196 raft_consensus.cc:513] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.647328 21391 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50642 (request call id 295) took 2256 ms. Trace:
I20250112 02:57:35.648702 22044 raft_consensus.cc:2463] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:35.650750 23109 catalog_manager.cc:1575] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: acquiring CA information for follower catalog manager: success
I20250112 02:57:35.560029 21387 rpcz_store.cc:276] 0112 02:57:32.723010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.240334 (+2517324us) service_pool.cc:225] Handling call
0112 02:57:35.425737 (+185403us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:35.427204 (+ 1467us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:35.462760 (+ 35556us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.462784 (+ 24us) write_op.cc:454] Released schema lock
0112 02:57:35.463017 (+ 233us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:35.559452 (+ 96435us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:35.559698 (+ 246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.430954 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.431348 (+ 394us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.431376 (+ 28us) write_op.cc:435] Acquired schema lock
0112 02:57:35.431393 (+ 17us) tablet.cc:659] Decoding operations
0112 02:57:35.431685 (+ 292us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.431732 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.431761 (+ 29us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:35.431986 (+ 225us) tablet.cc:698] Row locks acquired
0112 02:57:35.432002 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:35.432120 (+ 118us) write_op.cc:270] Start()
0112 02:57:35.432207 (+ 87us) write_op.cc:276] Timestamp: P: 1736650655432168 usec, L: 0
0112 02:57:35.432225 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.433033 (+ 808us) log.cc:844] Serialized 433 byte log entry
0112 02:57:35.442181 (+ 9148us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.442469 (+ 288us) write_op.cc:301] APPLY: starting
0112 02:57:35.442536 (+ 67us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.442631 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.442730 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.443079 (+ 349us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.443205 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:35.443246 (+ 41us) write_op.cc:312] APPLY: finished
0112 02:57:35.443402 (+ 156us) log.cc:844] Serialized 29 byte log entry
0112 02:57:35.443526 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.443587 (+ 61us) write_op.cc:454] Released schema lock
0112 02:57:35.443615 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":171,"apply.run_cpu_time_us":1300,"apply.run_wall_time_us":1299,"num_ops":2,"prepare.queue_time_us":2656,"prepare.run_cpu_time_us":2186,"prepare.run_wall_time_us":2406,"raft.queue_time_us":185,"raft.run_cpu_time_us":651,"raft.run_wall_time_us":651,"replication_time_us":9875}]]}
I20250112 02:57:35.652813 21387 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:34532 (request call id 1) took 1783 ms. Trace:
I20250112 02:57:35.653170 21387 rpcz_store.cc:276] 0112 02:57:33.869814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.652081 (+1782267us) service_pool.cc:225] Handling call
0112 02:57:35.652731 (+ 650us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:35.651108 21391 rpcz_store.cc:276] 0112 02:57:33.390861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.483238 (+2092377us) service_pool.cc:225] Handling call
0112 02:57:35.485928 (+ 2690us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:57:35.619376 (+133448us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:57:35.646753 (+ 27377us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:57:35.647032 (+ 279us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:57:35.647275 (+ 243us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.620547 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.620927 (+ 380us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.620951 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:35.620965 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:35.621239 (+ 274us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.621278 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.621303 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:35.621499 (+ 196us) tablet.cc:698] Row locks acquired
0112 02:57:35.621516 (+ 17us) write_op.cc:260] PREPARE: finished
0112 02:57:35.621626 (+ 110us) write_op.cc:270] Start()
0112 02:57:35.621721 (+ 95us) write_op.cc:276] Timestamp: P: 1736650655621755 usec, L: 0
0112 02:57:35.621736 (+ 15us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.622165 (+ 429us) log.cc:844] Serialized 433 byte log entry
0112 02:57:35.631098 (+ 8933us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.631384 (+ 286us) write_op.cc:301] APPLY: starting
0112 02:57:35.631435 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.631536 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.631634 (+ 98us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.632054 (+ 420us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.632200 (+ 146us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:35.632236 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:35.632384 (+ 148us) log.cc:844] Serialized 29 byte log entry
0112 02:57:35.632495 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.632549 (+ 54us) write_op.cc:454] Released schema lock
0112 02:57:35.632578 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:35.632781 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.632825 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"apply.run_cpu_time_us":1363,"apply.run_wall_time_us":1362,"num_ops":2,"prepare.queue_time_us":114,"prepare.run_cpu_time_us":1889,"prepare.run_wall_time_us":3991,"raft.queue_time_us":937,"raft.run_cpu_time_us":529,"raft.run_wall_time_us":529,"replication_time_us":9281}]]}
I20250112 02:57:35.655520 23178 ts_tablet_manager.cc:1939] T 443aa544f6df4d50a78c9effd306e216 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
W20250112 02:57:35.650799 21392 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41369773a0034c65b25fee871e6a25aa consensus state has no leader
I20250112 02:57:35.658561 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 443aa544f6df4d50a78c9effd306e216 (table default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]) successfully deleted
W20250112 02:57:35.677942 23095 log.cc:926] Time spent T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: Append to log took a long time: real 0.104s user 0.003s sys 0.000s
I20250112 02:57:35.602450 23022 raft_consensus.cc:513] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.682029 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:35.683281 21691 leader_election.cc:343] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.686897 23022 leader_election.cc:290] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:35.582275 21384 rpcz_store.cc:269] 0112 02:57:34.093811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:34.698087 (+604276us) service_pool.cc:225] Handling call
0112 02:57:34.849985 (+151898us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b54000415d4. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eed5b6d50 00007f7eed639f2a 00007f7eef2dedec 00007f7eef3d6dee 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc 00007f7eeb3fe236
0112 02:57:35.581790 (+731805us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":30,"spinlock_wait_cycles":136566016}
I20250112 02:57:35.695400 21392 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_6020a398e8e54211a4d2838ca624d5f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:35.696911 21392 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6020a398e8e54211a4d2838ca624d5f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:35.699890 21690 leader_election.cc:304] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.701417 23196 leader_election.cc:290] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.701896 22143 ts_tablet_manager.cc:1428] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.125s user 0.006s sys 0.001s
I20250112 02:57:35.702759 22143 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:35.703300 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:35.704453 21691 leader_election.cc:343] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.714433 22145 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:35.714854 22145 ts_tablet_manager.cc:1397] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.171s user 0.009s sys 0.039s
I20250112 02:57:35.717522 23196 raft_consensus.cc:491] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.717635 22145 raft_consensus.cc:357] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.718248 22145 raft_consensus.cc:383] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.717983 23196 raft_consensus.cc:513] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.718482 22145 raft_consensus.cc:738] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.719045 22145 consensus_queue.cc:260] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.719765 23196 leader_election.cc:290] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.721167 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.722275 21691 leader_election.cc:343] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.725900 23196 raft_consensus.cc:2798] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:35.726342 23196 raft_consensus.cc:695] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:35.727278 23196 consensus_queue.cc:237] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.741590 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:35.742251 22038 raft_consensus.cc:3054] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:35.745548 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet f445a4bf43db4571b6d3a4afe8fe823a because the tablet was not found. No further retry: Not found: Tablet not found: f445a4bf43db4571b6d3a4afe8fe823a
W20250112 02:57:35.747709 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5e3890251e644c1eb596de043b63491e because the tablet was not found. No further retry: Not found: Tablet not found: 5e3890251e644c1eb596de043b63491e
W20250112 02:57:35.749425 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 4e2b248224f9416faf4bd83a853ec918 because the tablet was not found. No further retry: Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:35.751060 22038 raft_consensus.cc:2463] T a69123c4ab384e0380c62e52b9f3b42d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:35.751060 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 15c16031f5b74cf9885fac110f445e1a because the tablet was not found. No further retry: Not found: Tablet not found: 15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:35.757295 22144 tablet_bootstrap.cc:492] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:35.757722 22144 ts_tablet_manager.cc:1397] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.189s user 0.020s sys 0.029s
I20250112 02:57:35.760344 22144 raft_consensus.cc:357] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.761045 22144 raft_consensus.cc:383] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.761317 22144 raft_consensus.cc:738] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.761977 22144 consensus_queue.cc:260] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.769976 21387 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50642:
name: "default.Table_f67a1fd5a8a0420db44f31cf1063241d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:57:35.773029 21387 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f67a1fd5a8a0420db44f31cf1063241d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:57:35.773396 22144 ts_tablet_manager.cc:1428] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20250112 02:57:35.774168 22144 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.776942 23124 raft_consensus.cc:513] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.788064 23124 leader_election.cc:290] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.789413 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:35.790619 21691 leader_election.cc:343] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.792757 23196 raft_consensus.cc:491] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.797487 23214 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:35.797308 21384 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50642:
table { table_name: "default.Table_00c60840930f46c49bf80fa7e4489419" } new_table_name: "default.Table_181a7bdabe584055b44baf7ff5ab440f" modify_external_catalogs: true
I20250112 02:57:35.798177 23214 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:35.801265 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:35.804212 22144 tablet_bootstrap.cc:654] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.793141 23196 raft_consensus.cc:513] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.813163 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.814429 21691 leader_election.cc:343] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.818748 23124 consensus_queue.cc:1035] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:35.826192 21899 raft_consensus.cc:3054] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.829046 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 21dc0755b38746a39bd773780e48cda5 (table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682]). Expected version 1 got 0
I20250112 02:57:35.830569 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
W20250112 02:57:35.834316 22114 log.cc:926] Time spent T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Append to log took a long time: real 0.197s user 0.041s sys 0.048s
I20250112 02:57:35.837240 21350 master-stress-test.cc:577] Killing test thread 1/10
I20250112 02:57:35.840304 22144 tablet_bootstrap.cc:492] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:35.840683 22144 ts_tablet_manager.cc:1397] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.005s
I20250112 02:57:35.756210 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716677cfad74c0dafe18c55796aeae5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:35.853464 22045 raft_consensus.cc:3054] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.854702 22145 ts_tablet_manager.cc:1428] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.140s user 0.003s sys 0.027s
I20250112 02:57:35.855532 22145 tablet_bootstrap.cc:492] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:35.757447 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.858049 22039 raft_consensus.cc:2463] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:35.859019 22144 raft_consensus.cc:357] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.859673 22144 raft_consensus.cc:383] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.859913 22144 raft_consensus.cc:738] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.860443 22144 consensus_queue.cc:260] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.764407 23168 raft_consensus.cc:491] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.861287 23168 raft_consensus.cc:513] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.819222 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:35.863371 22041 raft_consensus.cc:3054] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.863675 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.864769 21980 leader_election.cc:343] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.866698 23168 leader_election.cc:290] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:35.867810 22145 tablet_bootstrap.cc:654] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.870291 22144 ts_tablet_manager.cc:1428] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.029s user 0.005s sys 0.001s
I20250112 02:57:35.870321 22041 raft_consensus.cc:2463] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:35.871129 22144 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:35.876987 22144 tablet_bootstrap.cc:654] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:35.877336 23178 tablet_replica.cc:331] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:35.877164 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:35.860927 22045 raft_consensus.cc:2463] T 4716677cfad74c0dafe18c55796aeae5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:35.878785 21978 leader_election.cc:343] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.879223 21978 leader_election.cc:304] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.880882 23168 raft_consensus.cc:2743] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:35.887624 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 41369773a0034c65b25fee871e6a25aa (table default.Table_03e04d34d58e463e93d16e858752d3cb [id=30ad19fdfc1246ebbeaf73fff91f925e], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 41369773a0034c65b25fee871e6a25aa consensus state has no leader
I20250112 02:57:35.878388 23178 raft_consensus.cc:2238] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:35.889317 23178 raft_consensus.cc:2267] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:35.891423 23178 ts_tablet_manager.cc:1905] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:35.903707 22143 tablet_bootstrap.cc:654] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:35.905262 21691 consensus_peers.cc:487] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:35.919767 22143 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:35.920243 22143 ts_tablet_manager.cc:1397] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.218s user 0.010s sys 0.001s
I20250112 02:57:35.929474 22143 raft_consensus.cc:357] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.832104 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:35.930526 23190 raft_consensus.cc:491] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.931018 23190 raft_consensus.cc:513] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.931789 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]). Expected version 1 got 0
I20250112 02:57:35.933259 23190 leader_election.cc:290] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.933064 21390 catalog_manager.cc:5526] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:35.934098 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:35.934262 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:35.930254 22143 raft_consensus.cc:383] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:35.935297 21390 catalog_manager.cc:5526] T 8b6efb19f0cc4a96b957ec34c27ab836 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:35.937969 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:35.940047 21836 leader_election.cc:304] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:35.941020 21350 master-stress-test.cc:579] Killed test thread 1/10
I20250112 02:57:35.941206 23190 raft_consensus.cc:2743] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.941335 21350 master-stress-test.cc:577] Killing test thread 2/10
I20250112 02:57:35.941637 21350 master-stress-test.cc:579] Killed test thread 2/10
I20250112 02:57:35.941882 21350 master-stress-test.cc:577] Killing test thread 3/10
I20250112 02:57:35.942147 21350 master-stress-test.cc:579] Killed test thread 3/10
I20250112 02:57:35.942376 21350 master-stress-test.cc:577] Killing test thread 4/10
I20250112 02:57:35.942668 21350 master-stress-test.cc:579] Killed test thread 4/10
I20250112 02:57:35.942914 21350 master-stress-test.cc:577] Killing test thread 5/10
I20250112 02:57:35.948102 23196 leader_election.cc:290] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.935938 22143 raft_consensus.cc:738] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:35.953972 21690 leader_election.cc:304] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.954628 23126 raft_consensus.cc:2798] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:35.954936 23126 raft_consensus.cc:491] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:35.955243 23126 raft_consensus.cc:3054] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:35.955650 22143 consensus_queue.cc:260] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.961179 22143 ts_tablet_manager.cc:1428] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250112 02:57:35.963158 22143 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:35.961714 23126 raft_consensus.cc:513] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.963726 23192 raft_consensus.cc:491] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:35.965194 23192 raft_consensus.cc:513] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.966033 21690 leader_election.cc:304] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.966828 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:35.966995 21690 leader_election.cc:304] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.967103 23192 leader_election.cc:290] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.967650 23022 raft_consensus.cc:2798] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:35.968065 23022 raft_consensus.cc:695] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
W20250112 02:57:35.968175 21691 leader_election.cc:343] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.968307 23126 leader_election.cc:290] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:35.968886 23022 consensus_queue.cc:237] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.970212 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:35.971038 23168 raft_consensus.cc:491] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:35.971387 21691 leader_election.cc:343] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.971434 23168 raft_consensus.cc:513] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.967659 23192 raft_consensus.cc:2798] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:35.973045 23192 raft_consensus.cc:695] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:35.974285 23168 leader_election.cc:290] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:35.974043 23192 consensus_queue.cc:237] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:35.975236 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:35.976768 21978 leader_election.cc:343] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.981221 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:35.989037 21980 leader_election.cc:343] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:35.989418 21980 leader_election.cc:304] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:35.990248 23168 raft_consensus.cc:2743] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:35.990949 23214 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:35.992058 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:35.993502 22841 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:35.994678 23214 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:35.999228 22143 tablet_bootstrap.cc:654] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.008824 21690 leader_election.cc:304] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.009591 23022 raft_consensus.cc:2798] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.010025 23022 raft_consensus.cc:695] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:36.010768 23022 consensus_queue.cc:237] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.019218 21390 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:36.019897 21390 catalog_manager.cc:5526] T e4a9bc28dc5942608fb7224f9f058e27 P 1c2770701ca54288956df9d910d7d3d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20250112 02:57:36.021878 23192 consensus_queue.cc:1035] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:36.024969 22143 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:36.025065 22043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.025646 22043 raft_consensus.cc:2463] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:36.025815 22143 ts_tablet_manager.cc:1397] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.004s
I20250112 02:57:36.026738 23190 raft_consensus.cc:491] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.027153 23190 raft_consensus.cc:513] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.028980 23190 leader_election.cc:290] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.030349 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.030772 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.031976 22038 raft_consensus.cc:2463] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:36.032807 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:36.033385 22041 raft_consensus.cc:3054] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.036167 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f8052651fa496cbb3702c17f592106" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.037127 22144 tablet_bootstrap.cc:492] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:36.037521 22144 ts_tablet_manager.cc:1397] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.015s
I20250112 02:57:36.044947 22145 tablet_bootstrap.cc:492] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:36.045354 22145 ts_tablet_manager.cc:1397] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.190s user 0.014s sys 0.019s
I20250112 02:57:36.048790 23186 ts_tablet_manager.cc:1918] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:36.049261 23186 log.cc:1198] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:36.050091 23186 ts_tablet_manager.cc:1939] T 84f7700b8cce4261a32f2009d1481b99 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:36.051784 23190 raft_consensus.cc:491] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.052209 23190 raft_consensus.cc:513] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.054960 23190 leader_election.cc:290] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.056192 22041 raft_consensus.cc:2463] T 66fe67a814504a89b6fa8f46e2a4aee9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:36.057186 21836 leader_election.cc:304] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:36.062428 23190 raft_consensus.cc:2743] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.064397 22144 raft_consensus.cc:357] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.065002 22144 raft_consensus.cc:383] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.065289 22144 raft_consensus.cc:738] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.074558 21392 master_service.cc:946] ReplaceTablet: received request to replace tablet a4b543ed18b34cd58a1c588ed7d079c5 from {username='slave'} at 127.0.0.1:50642
I20250112 02:57:36.058772 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.055353 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.070248 22143 raft_consensus.cc:357] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.093703 22143 raft_consensus.cc:383] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.093963 22143 raft_consensus.cc:738] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.094671 21835 leader_election.cc:304] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:36.094485 22143 consensus_queue.cc:260] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.095315 23190 raft_consensus.cc:2743] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.096449 22143 ts_tablet_manager.cc:1428] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20250112 02:57:36.097445 22143 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:36.097546 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 272) took 1008 ms. Trace:
I20250112 02:57:36.068668 22145 raft_consensus.cc:357] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.109925 22145 raft_consensus.cc:383] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.110352 22145 raft_consensus.cc:738] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.111027 22145 consensus_queue.cc:260] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.112933 22145 ts_tablet_manager.cc:1428] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.067s user 0.006s sys 0.000s
I20250112 02:57:36.131930 23109 catalog_manager.cc:1603] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:57:36.134706 23184 ts_tablet_manager.cc:1918] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:36.135144 23184 log.cc:1198] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/96f98f6840e24f3c9d03937c1bbc00e7
I20250112 02:57:36.135921 23184 ts_tablet_manager.cc:1939] T 96f98f6840e24f3c9d03937c1bbc00e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:36.148382 21392 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet a4b543ed18b34cd58a1c588ed7d079c5
I20250112 02:57:36.149296 22145 tablet_bootstrap.cc:492] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:36.151549 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 95f62433f15a43eb9a654c067052cd68) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:36.152261 21392 catalog_manager.cc:6596] ReplaceTablet: tablet a4b543ed18b34cd58a1c588ed7d079c5 deleted and replaced by tablet 95f62433f15a43eb9a654c067052cd68
I20250112 02:57:36.150413 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 95f62433f15a43eb9a654c067052cd68) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:36.169065 22145 tablet_bootstrap.cc:654] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.175194 21949 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:36.178427 23215 raft_consensus.cc:491] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.178895 23215 raft_consensus.cc:513] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.181448 23215 leader_election.cc:290] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.182039 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:36.183287 21980 leader_election.cc:343] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.065963 22144 consensus_queue.cc:260] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.185714 23190 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.186100 23190 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.187112 22144 ts_tablet_manager.cc:1428] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.149s user 0.006s sys 0.001s
I20250112 02:57:36.188278 23190 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:36.195717 21836 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 443aa544f6df4d50a78c9effd306e216
I20250112 02:57:36.195938 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 253) took 1131 ms. Trace:
I20250112 02:57:36.196302 23184 rpcz_store.cc:276] 0112 02:57:35.064195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.083370 (+ 19175us) service_pool.cc:225] Handling call
0112 02:57:35.640560 (+557190us) ts_tablet_manager.cc:1145] Deleting tablet 96f98f6840e24f3c9d03937c1bbc00e7
0112 02:57:35.640608 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:35.960451 (+319843us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.134467 (+174016us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.195866 (+ 61399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":556567}
I20250112 02:57:36.197610 23184 tablet_replica.cc:331] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:36.199065 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 96f98f6840e24f3c9d03937c1bbc00e7 (table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]) successfully deleted
I20250112 02:57:36.199744 23184 raft_consensus.cc:2238] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:36.200407 23184 raft_consensus.cc:2267] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.107088 22143 tablet_bootstrap.cc:654] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.203840 23186 rpcz_store.cc:276] 0112 02:57:35.088771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.088918 (+ 147us) service_pool.cc:225] Handling call
0112 02:57:35.355317 (+266399us) ts_tablet_manager.cc:1145] Deleting tablet 84f7700b8cce4261a32f2009d1481b99
0112 02:57:35.355353 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:35.832911 (+477558us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.048546 (+215635us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.097492 (+ 48946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":265866}
I20250112 02:57:36.205065 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 84f7700b8cce4261a32f2009d1481b99 (table default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]) successfully deleted
I20250112 02:57:36.092124 21690 leader_election.cc:304] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:36.206480 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 114504us
I20250112 02:57:36.207301 21690 leader_election.cc:304] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.208536 21690 leader_election.cc:304] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.208801 23215 raft_consensus.cc:491] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.209347 23215 raft_consensus.cc:513] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.150305 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 95f62433f15a43eb9a654c067052cd68) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:36.182422 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.194178 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.211248 21754 raft_consensus.cc:2388] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:36.212152 23215 leader_election.cc:290] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.212179 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:36.212422 21835 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:36.212692 23126 raft_consensus.cc:2798] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:36.213013 23126 raft_consensus.cc:491] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:57:36.213150 21978 leader_election.cc:343] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.213312 23126 raft_consensus.cc:3054] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.213543 23190 raft_consensus.cc:3054] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.213505 21978 leader_election.cc:304] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.214403 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.214763 23169 raft_consensus.cc:2743] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:36.215471 21978 leader_election.cc:343] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.216753 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 274) took 1126 ms. Trace:
I20250112 02:57:36.217105 23186 rpcz_store.cc:276] 0112 02:57:35.090161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.090276 (+ 115us) service_pool.cc:225] Handling call
0112 02:57:36.216505 (+1126229us) ts_tablet_manager.cc:1145] Deleting tablet 994e1d114e5a4de5aa1d7651907d718e
0112 02:57:36.216547 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.216700 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1125711}
I20250112 02:57:36.217833 23194 raft_consensus.cc:2798] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250112 02:57:36.218092 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 994e1d114e5a4de5aa1d7651907d718e because the tablet was not found. No further retry: Not found: Tablet not found: 994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:36.218122 23194 raft_consensus.cc:491] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:36.218436 23194 raft_consensus.cc:3054] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.218840 23196 raft_consensus.cc:2798] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.219285 23196 raft_consensus.cc:695] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:36.220054 23196 consensus_queue.cc:237] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:36.223877 21980 leader_election.cc:343] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.224244 21980 leader_election.cc:304] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.224427 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 273) took 1134 ms. Trace:
I20250112 02:57:36.224853 23215 raft_consensus.cc:2743] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.226034 21899 raft_consensus.cc:3054] T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.227999 21901 raft_consensus.cc:3054] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.233412 23194 raft_consensus.cc:513] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.203747 23184 ts_tablet_manager.cc:1905] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.246578 23194 leader_election.cc:290] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.248580 22144 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:36.251080 23190 raft_consensus.cc:2743] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.237095 23126 raft_consensus.cc:513] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.253002 22145 tablet_bootstrap.cc:492] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:36.253392 22145 ts_tablet_manager.cc:1397] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.104s user 0.003s sys 0.008s
I20250112 02:57:36.253691 23126 leader_election.cc:290] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.258785 22145 raft_consensus.cc:357] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.259371 23178 ts_tablet_manager.cc:1918] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:36.259428 22145 raft_consensus.cc:383] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.259752 22145 raft_consensus.cc:738] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.260354 22145 consensus_queue.cc:260] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.267212 23240 tablet.cc:1721] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:36.267822 21872 tablet_service.cc:1467] Processing CreateTablet for tablet 4494e809c9ee4a0884da38b45e7a90b2 (DEFAULT_TABLE table=default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:36.268790 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4494e809c9ee4a0884da38b45e7a90b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:36.270104 22145 ts_tablet_manager.cc:1428] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.016s user 0.001s sys 0.006s
I20250112 02:57:36.270920 22145 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:36.259776 23178 log.cc:1198] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/5bd0d70fb0fe4a179b1ea6fa23d180c9
I20250112 02:57:36.272275 23178 ts_tablet_manager.cc:1939] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:36.274947 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 (table default.Table_102d4e698b854093aa5caa6c1e25ad50 [id=49e7e77e80814c8f9128ea9878a6eb21]) successfully deleted
I20250112 02:57:36.276960 23178 tablet_replica.cc:331] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:36.277763 23178 raft_consensus.cc:2238] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:36.280047 22143 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:36.280473 22143 ts_tablet_manager.cc:1397] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.183s user 0.008s sys 0.007s
I20250112 02:57:36.282621 22143 raft_consensus.cc:357] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.283263 22143 raft_consensus.cc:383] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.283507 22143 raft_consensus.cc:738] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.283751 23178 pending_rounds.cc:62] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Trying to abort 1 pending ops.
I20250112 02:57:36.284019 22143 consensus_queue.cc:260] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.284579 23178 pending_rounds.cc:69] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321082981859328 op_type: NO_OP noop_request { }
I20250112 02:57:36.284924 23178 raft_consensus.cc:2883] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:57:36.285220 23178 raft_consensus.cc:2267] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.285898 22143 ts_tablet_manager.cc:1428] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20250112 02:57:36.286634 22145 tablet_bootstrap.cc:654] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.290745 23178 ts_tablet_manager.cc:1905] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.296175 21388 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 03d2021384bb41b58a66fcb7afaeaca8
I20250112 02:57:36.298226 21388 catalog_manager.cc:6596] ReplaceTablet: tablet 03d2021384bb41b58a66fcb7afaeaca8 deleted and replaced by tablet 3fef7ac20f034f589e564e5f5e754705
I20250112 02:57:36.298660 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fef7ac20f034f589e564e5f5e754705) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:36.224732 23186 rpcz_store.cc:276] 0112 02:57:35.089820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.090298 (+ 478us) service_pool.cc:225] Handling call
0112 02:57:36.224178 (+1133880us) ts_tablet_manager.cc:1145] Deleting tablet 5e3890251e644c1eb596de043b63491e
0112 02:57:36.224223 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.224375 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133238}
I20250112 02:57:36.299566 21388 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:50642 (request call id 294) took 3086 ms. Trace:
I20250112 02:57:36.299937 21388 rpcz_store.cc:276] 0112 02:57:33.213207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.480751 (+2267544us) service_pool.cc:225] Handling call
0112 02:57:36.299408 (+818657us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:36.282120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:36.282523 (+ 403us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:36.282563 (+ 40us) write_op.cc:435] Acquired schema lock
0112 02:57:36.282584 (+ 21us) tablet.cc:659] Decoding operations
0112 02:57:36.282918 (+ 334us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:36.282961 (+ 43us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:36.282985 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:36.283286 (+ 301us) tablet.cc:698] Row locks acquired
0112 02:57:36.283299 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:36.283422 (+ 123us) write_op.cc:270] Start()
0112 02:57:36.283556 (+ 134us) write_op.cc:276] Timestamp: P: 1736650656283406 usec, L: 0
0112 02:57:36.283574 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:36.284210 (+ 636us) log.cc:844] Serialized 631 byte log entry
0112 02:57:36.293241 (+ 9031us) op_driver.cc:464] REPLICATION: finished
0112 02:57:36.293538 (+ 297us) write_op.cc:301] APPLY: starting
0112 02:57:36.293590 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:36.293689 (+ 99us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:36.293785 (+ 96us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:36.294105 (+ 320us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:36.294295 (+ 190us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:36.294331 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:36.294483 (+ 152us) log.cc:844] Serialized 29 byte log entry
0112 02:57:36.294629 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.294701 (+ 72us) write_op.cc:454] Released schema lock
0112 02:57:36.294733 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:36.294975 (+ 242us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.295002 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":170,"apply.run_cpu_time_us":1381,"apply.run_wall_time_us":1380,"num_ops":2,"prepare.queue_time_us":1029,"prepare.run_cpu_time_us":2267,"prepare.run_wall_time_us":4373,"raft.queue_time_us":126,"raft.run_cpu_time_us":635,"raft.run_wall_time_us":635,"replication_time_us":9585}]]}
I20250112 02:57:36.303903 21882 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fef7ac20f034f589e564e5f5e754705) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:36.304948 22145 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:36.305327 22145 ts_tablet_manager.cc:1397] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.007s
I20250112 02:57:36.314610 22145 raft_consensus.cc:357] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.315253 22145 raft_consensus.cc:383] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.315532 22145 raft_consensus.cc:738] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.329401 22145 consensus_queue.cc:260] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.338929 22145 ts_tablet_manager.cc:1428] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.033s user 0.003s sys 0.004s
I20250112 02:57:36.339741 22145 tablet_bootstrap.cc:492] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:36.340090 22144 tablet_bootstrap.cc:654] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.349705 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:36.351169 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:36.354604 21389 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]). Expected version 1 got 0
I20250112 02:57:36.356830 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 4494e809c9ee4a0884da38b45e7a90b2 (DEFAULT_TABLE table=default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:36.361274 22145 tablet_bootstrap.cc:654] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.379168 23235 raft_consensus.cc:491] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.379671 23235 raft_consensus.cc:513] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.380142 23178 ts_tablet_manager.cc:1918] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:36.380523 23178 log.cc:1198] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/a4b543ed18b34cd58a1c588ed7d079c5
I20250112 02:57:36.381212 23178 ts_tablet_manager.cc:1939] T a4b543ed18b34cd58a1c588ed7d079c5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:36.381680 23235 leader_election.cc:290] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.383441 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4494e809c9ee4a0884da38b45e7a90b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:36.391567 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:36.392727 21980 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.340590 23184 ts_tablet_manager.cc:1918] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:36.397608 23184 log.cc:1198] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/994e1d114e5a4de5aa1d7651907d718e
I20250112 02:57:36.398278 23184 ts_tablet_manager.cc:1939] T 994e1d114e5a4de5aa1d7651907d718e P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:36.400269 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 252) took 1336 ms. Trace:
I20250112 02:57:36.400657 23184 rpcz_store.cc:276] 0112 02:57:35.063896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.107870 (+ 43974us) service_pool.cc:225] Handling call
0112 02:57:36.197481 (+1089611us) ts_tablet_manager.cc:1145] Deleting tablet 994e1d114e5a4de5aa1d7651907d718e
0112 02:57:36.197516 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.291665 (+ 94149us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.340371 (+ 48706us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.400202 (+ 59831us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089056}
I20250112 02:57:36.401883 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 994e1d114e5a4de5aa1d7651907d718e (table default.Table_202c2f292aed4f11a310b5cf5c271aeb [id=c6af6d885ff34aa294d83e9859c17a4e]) successfully deleted
I20250112 02:57:36.402871 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 258) took 1265 ms. Trace:
I20250112 02:57:36.403203 23184 rpcz_store.cc:276] 0112 02:57:35.137334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.137488 (+ 154us) service_pool.cc:225] Handling call
0112 02:57:36.402604 (+1265116us) ts_tablet_manager.cc:1145] Deleting tablet 9543a0a0359e4b72bd30f1cb6290f767
0112 02:57:36.402647 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.402816 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264346}
W20250112 02:57:36.404373 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 9543a0a0359e4b72bd30f1cb6290f767 because the tablet was not found. No further retry: Not found: Tablet not found: 9543a0a0359e4b72bd30f1cb6290f767
I20250112 02:57:36.405301 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 257) took 1269 ms. Trace:
I20250112 02:57:36.405648 23184 rpcz_store.cc:276] 0112 02:57:35.135610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.143759 (+ 8149us) service_pool.cc:225] Handling call
0112 02:57:36.405026 (+1261267us) ts_tablet_manager.cc:1145] Deleting tablet 98aa732d27f144aaad088ecc072837e9
0112 02:57:36.405071 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.405247 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260598}
W20250112 02:57:36.406757 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 98aa732d27f144aaad088ecc072837e9 because the tablet was not found. No further retry: Not found: Tablet not found: 98aa732d27f144aaad088ecc072837e9
W20250112 02:57:36.419909 23238 log.cc:926] Time spent T e4a9bc28dc5942608fb7224f9f058e27 P cca160b7cffb4ae287ea077e7352ffc9: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
I20250112 02:57:36.427556 23169 raft_consensus.cc:491] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.427989 23169 raft_consensus.cc:513] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.429839 23169 leader_election.cc:290] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.431491 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:36.432677 21980 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:36.434764 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 5e3890251e644c1eb596de043b63491e because the tablet was not found. No further retry: Not found: Tablet not found: 5e3890251e644c1eb596de043b63491e
I20250112 02:57:36.435194 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.436039 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet a4b543ed18b34cd58a1c588ed7d079c5 (table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]) successfully deleted
I20250112 02:57:36.438966 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.440881 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:36.442239 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2021384bb41b58a66fcb7afaeaca8 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fef7ac20f034f589e564e5f5e754705) from {username='slave'} at 127.0.0.1:46754
W20250112 02:57:36.443634 21691 leader_election.cc:343] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.444247 21727 tablet_service.cc:1467] Processing CreateTablet for tablet 4494e809c9ee4a0884da38b45e7a90b2 (DEFAULT_TABLE table=default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:57:36.444614 21978 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.445004 21978 leader_election.cc:304] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:36.446043 21978 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:36.446755 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 237342us
I20250112 02:57:36.447647 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:36.448709 23169 raft_consensus.cc:491] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.449155 23169 raft_consensus.cc:513] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.450016 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 21dc0755b38746a39bd773780e48cda5 (table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682]). Expected version 1 got 0
I20250112 02:57:36.450742 23215 raft_consensus.cc:2743] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.451228 23215 raft_consensus.cc:491] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.451566 23215 raft_consensus.cc:513] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.453298 23215 leader_election.cc:290] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.454396 23169 leader_election.cc:290] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.446415 21978 leader_election.cc:304] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.456241 22143 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:36.456449 23235 raft_consensus.cc:2743] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.460045 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1144ac759a4db496f67cdc9ec35020" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:36.460402 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4494e809c9ee4a0884da38b45e7a90b2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:36.461303 21980 leader_election.cc:343] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.463321 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:36.464653 21980 leader_election.cc:343] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.465152 23235 raft_consensus.cc:491] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.465616 23235 raft_consensus.cc:513] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.467869 23235 leader_election.cc:290] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.470399 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:36.471612 21980 leader_election.cc:343] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.496544 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:36.497081 22044 raft_consensus.cc:3054] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.497282 22145 tablet_bootstrap.cc:492] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:36.497682 22145 ts_tablet_manager.cc:1397] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.008s
I20250112 02:57:36.499924 22143 tablet_bootstrap.cc:654] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.501560 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:36.502148 22040 raft_consensus.cc:3054] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.505424 23224 raft_consensus.cc:491] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.505851 23224 raft_consensus.cc:513] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.507273 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:36.507803 23224 leader_election.cc:290] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.508839 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
W20250112 02:57:36.512531 21835 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:36.516103 23186 tablet_replica.cc:331] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:36.523720 22143 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:36.524118 22143 ts_tablet_manager.cc:1397] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.005s
I20250112 02:57:36.527453 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 256) took 1392 ms. Trace:
I20250112 02:57:36.527837 23184 rpcz_store.cc:276] 0112 02:57:35.135279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.145368 (+ 10089us) service_pool.cc:225] Handling call
0112 02:57:36.478154 (+1332786us) ts_tablet_manager.cc:1145] Deleting tablet c48d1bc147ba48ebbebfd707dc44fe02
0112 02:57:36.527174 (+ 49020us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.527387 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1332206}
W20250112 02:57:36.529129 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet c48d1bc147ba48ebbebfd707dc44fe02 because the tablet was not found. No further retry: Not found: Tablet not found: c48d1bc147ba48ebbebfd707dc44fe02
I20250112 02:57:36.531586 22145 raft_consensus.cc:357] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.532338 22145 raft_consensus.cc:383] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.532594 22145 raft_consensus.cc:738] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.535924 22040 raft_consensus.cc:2463] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:36.536717 23184 tablet_replica.cc:331] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
W20250112 02:57:36.537634 21836 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:36.518309 23186 raft_consensus.cc:2238] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:36.538009 21836 leader_election.cc:304] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:36.539072 23190 raft_consensus.cc:2743] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:36.539446 21377 catalog_manager.cc:4852] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 03d2021384bb41b58a66fcb7afaeaca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d2021384bb41b58a66fcb7afaeaca8 already in progress: creating tablet
I20250112 02:57:36.534636 22143 raft_consensus.cc:357] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.539913 22143 raft_consensus.cc:383] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.540186 22143 raft_consensus.cc:738] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.540504 23184 raft_consensus.cc:2238] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:36.541016 23184 raft_consensus.cc:2267] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.540776 22143 consensus_queue.cc:260] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.543069 23184 ts_tablet_manager.cc:1905] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.545392 23186 raft_consensus.cc:2267] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.548343 23186 ts_tablet_manager.cc:1905] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.552273 22143 ts_tablet_manager.cc:1428] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250112 02:57:36.553627 22143 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:36.560400 21690 consensus_peers.cc:487] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4b543ed18b34cd58a1c588ed7d079c5. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:36.562575 21690 leader_election.cc:304] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.563325 23194 raft_consensus.cc:2798] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.563916 23194 raft_consensus.cc:695] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:36.564985 22044 raft_consensus.cc:2463] T ae3ed0bdc2ec44998ee461d4b1254e91 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:36.564998 23194 consensus_queue.cc:237] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.578059 21690 leader_election.cc:304] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:36.578960 23246 raft_consensus.cc:2798] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.579617 23246 raft_consensus.cc:695] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:36.580432 23246 consensus_queue.cc:237] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.601282 22144 tablet_bootstrap.cc:492] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:36.601691 22144 ts_tablet_manager.cc:1397] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.353s user 0.015s sys 0.002s
I20250112 02:57:36.603892 22144 raft_consensus.cc:357] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.604528 22144 raft_consensus.cc:383] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.604776 22144 raft_consensus.cc:738] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.622848 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.623062 22145 consensus_queue.cc:260] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.623448 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.624526 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1144ac759a4db496f67cdc9ec35020" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.625239 22145 ts_tablet_manager.cc:1428] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.127s user 0.005s sys 0.001s
I20250112 02:57:36.626085 22145 tablet_bootstrap.cc:492] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:36.630786 22144 consensus_queue.cc:260] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.634603 22143 tablet_bootstrap.cc:654] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.510376 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]). Expected version 1 got 0
I20250112 02:57:36.635607 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:36.636236 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:36.636912 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:36.637375 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]). Expected version 1 got 0
I20250112 02:57:36.638945 23242 raft_consensus.cc:491] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.639397 23242 raft_consensus.cc:513] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.641436 23242 leader_election.cc:290] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:36.638579 21691 leader_election.cc:343] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.644536 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.647423 23194 raft_consensus.cc:491] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.647907 23194 raft_consensus.cc:513] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.650615 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.650830 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:36.651221 22040 raft_consensus.cc:2463] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:36.651852 21836 leader_election.cc:304] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:36.652338 21690 leader_election.cc:304] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:36.653009 22144 ts_tablet_manager.cc:1428] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.051s user 0.006s sys 0.001s
I20250112 02:57:36.653645 23192 raft_consensus.cc:2798] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:36.653789 22144 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:36.653937 23192 raft_consensus.cc:491] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:36.654224 23192 raft_consensus.cc:3054] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.654619 23242 raft_consensus.cc:2743] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.661456 23192 raft_consensus.cc:513] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.668704 22144 tablet_bootstrap.cc:654] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.670269 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:36.671027 22044 raft_consensus.cc:3054] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.674615 23194 leader_election.cc:290] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.687685 23235 raft_consensus.cc:491] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.688164 23235 raft_consensus.cc:513] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.689889 23235 leader_election.cc:290] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.691947 22145 tablet_bootstrap.cc:654] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.692245 23192 leader_election.cc:290] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:36.693225 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:36.638096 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 13edf2c56e8e4254b75e0eb85fb88427 (table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]). Expected version 1 got 0
I20250112 02:57:36.699383 21383 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 685953572d45476ebaf2580fd5e4a3cd (table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]). Expected version 1 got 0
I20250112 02:57:36.701503 23184 ts_tablet_manager.cc:1918] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:36.702471 21899 raft_consensus.cc:3054] T 2722e4001ba44a7998f4ba7682bd3c44 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.702666 23184 log.cc:1198] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:36.703563 23184 ts_tablet_manager.cc:1939] T 84f7700b8cce4261a32f2009d1481b99 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:36.719959 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:36.720543 21754 raft_consensus.cc:2463] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
I20250112 02:57:36.730032 22143 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:36.730432 22143 ts_tablet_manager.cc:1397] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.177s user 0.007s sys 0.002s
I20250112 02:57:36.717047 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:36.718405 21902 raft_consensus.cc:3054] T 8b6efb19f0cc4a96b957ec34c27ab836 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.739264 22143 raft_consensus.cc:357] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.739867 22143 raft_consensus.cc:383] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.740097 22143 raft_consensus.cc:738] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:36.694123 21980 leader_election.cc:343] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:36.740684 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 46620.9us
I20250112 02:57:36.718762 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78f03a1dfb604b33a0528c28ade287f2" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:36.728200 22144 tablet_bootstrap.cc:492] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:36.742774 22144 ts_tablet_manager.cc:1397] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.006s
I20250112 02:57:36.744951 22144 raft_consensus.cc:357] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.746117 22144 raft_consensus.cc:383] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.746366 22144 raft_consensus.cc:738] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.747100 22144 consensus_queue.cc:260] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.696630 22044 raft_consensus.cc:2463] T 78f03a1dfb604b33a0528c28ade287f2 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
W20250112 02:57:36.700922 21978 leader_election.cc:343] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.753253 21978 leader_election.cc:304] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:36.754081 21978 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 53206.3us
W20250112 02:57:36.755218 21978 leader_election.cc:343] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.755614 21978 leader_election.cc:304] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.756282 23169 raft_consensus.cc:2743] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.756819 23169 raft_consensus.cc:2743] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:36.757684 21978 leader_election.cc:343] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.757936 22144 ts_tablet_manager.cc:1428] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.015s user 0.003s sys 0.002s
I20250112 02:57:36.758018 21978 leader_election.cc:304] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.758774 22144 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:36.758848 23235 raft_consensus.cc:2743] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:36.755398 23184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 259) took 1173 ms. Trace:
I20250112 02:57:36.761592 21690 leader_election.cc:304] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:36.761749 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:36.762295 23192 raft_consensus.cc:2798] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.762923 23192 raft_consensus.cc:695] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:36.763723 23192 consensus_queue.cc:237] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.767699 23186 ts_tablet_manager.cc:1918] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:36.767621 23184 rpcz_store.cc:276] 0112 02:57:35.582104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.582229 (+ 125us) service_pool.cc:225] Handling call
0112 02:57:36.529937 (+947708us) ts_tablet_manager.cc:1145] Deleting tablet 84f7700b8cce4261a32f2009d1481b99
0112 02:57:36.536530 (+ 6593us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.602695 (+ 66165us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.701291 (+ 98596us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.755334 (+ 54043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":947209}
I20250112 02:57:36.768929 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:36.771129 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 13edf2c56e8e4254b75e0eb85fb88427 (table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]). Expected version 1 got 0
I20250112 02:57:36.778051 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet e7d8db14a8054ae69daf42dd76f35511 (table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718]). Expected version 1 got 0
I20250112 02:57:36.779350 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:36.780493 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:36.785488 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 685953572d45476ebaf2580fd5e4a3cd (table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]). Expected version 1 got 0
I20250112 02:57:36.786031 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]). Expected version 1 got 0
I20250112 02:57:36.787689 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:36.788304 23186 log.cc:1198] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:36.788383 21387 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740]). Expected version 1 got 0
W20250112 02:57:36.769204 21691 leader_election.cc:343] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:36.790589 21691 leader_election.cc:343] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.790987 21386 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 247f0ef0ada44d9d957c8ff42b0da4b5
I20250112 02:57:36.792387 21386 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 293) took 3591 ms. Trace:
I20250112 02:57:36.796355 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:36.797026 23178 tablet_replica.cc:331] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:36.792662 21386 rpcz_store.cc:276] 0112 02:57:33.200726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.401568 (+2200842us) service_pool.cc:225] Handling call
0112 02:57:35.571065 (+169497us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:35.571565 (+ 500us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:35.571708 (+ 143us) catalog_manager.cc:2762] Locking tablets
0112 02:57:36.309948 (+738240us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:36.327926 (+ 17978us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:36.328152 (+ 226us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:36.790877 (+462725us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:36.792331 (+ 1454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:36.313886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:36.314282 (+ 396us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:36.314309 (+ 27us) write_op.cc:435] Acquired schema lock
0112 02:57:36.314323 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:36.314688 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:36.314734 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:36.314756 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:36.315301 (+ 545us) tablet.cc:698] Row locks acquired
0112 02:57:36.315322 (+ 21us) write_op.cc:260] PREPARE: finished
0112 02:57:36.315447 (+ 125us) write_op.cc:270] Start()
0112 02:57:36.315692 (+ 245us) write_op.cc:276] Timestamp: P: 1736650656315444 usec, L: 0
0112 02:57:36.315713 (+ 21us) op_driver.cc:348] REPLICATION: starting
0112 02:57:36.316312 (+ 599us) log.cc:844] Serialized 734 byte log entry
0112 02:57:36.326177 (+ 9865us) op_driver.cc:464] REPLICATION: finished
0112 02:57:36.326563 (+ 386us) write_op.cc:301] APPLY: starting
0112 02:57:36.326617 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:36.326721 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:36.326809 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:36.327063 (+ 254us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:36.327206 (+ 143us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:36.327244 (+ 38us) write_op.cc:312] APPLY: finished
0112 02:57:36.327410 (+ 166us) log.cc:844] Serialized 29 byte log entry
0112 02:57:36.327542 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.327604 (+ 62us) write_op.cc:454] Released schema lock
0112 02:57:36.327634 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:36.329222 (+ 1588us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.329249 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":234,"apply.run_cpu_time_us":1237,"apply.run_wall_time_us":1235,"num_ops":2,"prepare.queue_time_us":2282,"prepare.run_cpu_time_us":2342,"prepare.run_wall_time_us":15325,"raft.queue_time_us":80,"raft.run_cpu_time_us":639,"raft.run_wall_time_us":639,"replication_time_us":10373}]]}
I20250112 02:57:36.797987 23178 raft_consensus.cc:2238] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:36.798440 23178 raft_consensus.cc:2267] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.781605 22144 tablet_bootstrap.cc:654] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.800391 23178 ts_tablet_manager.cc:1905] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.800246 23240 tablet.cc:1721] T 65eb8c99917247edbdbf5de42f3b505c P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:36.795991 21872 tablet_service.cc:1514] Processing DeleteTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:36.806067 21978 leader_election.cc:304] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.806797 23169 raft_consensus.cc:2798] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:36.807142 23169 raft_consensus.cc:491] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:36.807446 23169 raft_consensus.cc:3054] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.808346 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 84f7700b8cce4261a32f2009d1481b99 (table default.Table_a1ba8d2b677e47f5bffefd5fa58046e4 [id=ecfa4990387d41d5ac1d8bb7b790e1b5]) successfully deleted
I20250112 02:57:36.809392 23184 tablet_replica.cc:331] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:36.810192 23184 raft_consensus.cc:2238] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:36.814375 23169 raft_consensus.cc:513] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.817461 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:36.818733 21980 leader_election.cc:343] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:36.770944 22143 consensus_queue.cc:260] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.821257 22143 ts_tablet_manager.cc:1428] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.091s user 0.004s sys 0.000s
I20250112 02:57:36.822818 22144 tablet_bootstrap.cc:492] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:36.823210 22144 ts_tablet_manager.cc:1397] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.065s user 0.006s sys 0.008s
I20250112 02:57:36.829638 22145 tablet_bootstrap.cc:492] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:36.830950 22144 raft_consensus.cc:357] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.831573 22144 raft_consensus.cc:383] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.831787 22144 raft_consensus.cc:738] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.832268 22144 consensus_queue.cc:260] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.834426 22144 ts_tablet_manager.cc:1428] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250112 02:57:36.835214 22144 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:36.837322 23184 raft_consensus.cc:2267] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:36.839372 23184 ts_tablet_manager.cc:1905] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:36.852340 22145 ts_tablet_manager.cc:1397] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.226s user 0.005s sys 0.005s
I20250112 02:57:36.826251 23186 ts_tablet_manager.cc:1939] T f445a4bf43db4571b6d3a4afe8fe823a P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
W20250112 02:57:36.846746 21690 consensus_peers.cc:487] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bd0d70fb0fe4a179b1ea6fa23d180c9. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:36.853045 21727 tablet_service.cc:1514] Processing DeleteTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:46754
I20250112 02:57:36.858649 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:36.859119 21755 raft_consensus.cc:3054] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:36.864547 22143 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:36.868310 21755 raft_consensus.cc:2463] T 6e922b68d33b4a76a260816475ee6a3c P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:36.869436 22145 raft_consensus.cc:357] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.870241 22145 raft_consensus.cc:383] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:36.873270 21691 consensus_peers.cc:487] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:36.873826 22144 tablet_bootstrap.cc:654] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:36.874200 21691 consensus_peers.cc:487] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:36.907305 23192 consensus_queue.cc:1035] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:36.831828 23169 leader_election.cc:290] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.830844 23178 ts_tablet_manager.cc:1918] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:36.871996 23261 tablet.cc:1721] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:36.894858 21978 leader_election.cc:304] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:36.911355 23178 log.cc:1198] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/247f0ef0ada44d9d957c8ff42b0da4b5
I20250112 02:57:36.912030 23178 ts_tablet_manager.cc:1939] T 247f0ef0ada44d9d957c8ff42b0da4b5 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:36.912400 23235 raft_consensus.cc:2798] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:36.912837 23235 raft_consensus.cc:695] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:36.913601 23235 consensus_queue.cc:237] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.914419 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 247f0ef0ada44d9d957c8ff42b0da4b5 (table default.Table_ff93d70a357e442dafed35cd75ae6152 [id=7747b35d16654e1188fa4474b6e55dce]) successfully deleted
I20250112 02:57:36.912665 22145 raft_consensus.cc:738] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.919085 22145 consensus_queue.cc:260] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.921672 22145 ts_tablet_manager.cc:1428] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.054s user 0.000s sys 0.004s
I20250112 02:57:36.922644 22145 tablet_bootstrap.cc:492] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:36.930341 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 276) took 1819 ms. Trace:
I20250112 02:57:36.931133 23186 rpcz_store.cc:276] 0112 02:57:35.111390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.113878 (+ 2488us) service_pool.cc:225] Handling call
0112 02:57:36.299729 (+1185851us) ts_tablet_manager.cc:1145] Deleting tablet f445a4bf43db4571b6d3a4afe8fe823a
0112 02:57:36.515657 (+215928us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:36.600313 (+ 84656us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.767477 (+167164us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:36.930287 (+162810us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69778,"tablet-delete.queue_time_us":1185312}
I20250112 02:57:36.945030 21388 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:36.946413 21388 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:36.960104 22144 tablet_bootstrap.cc:492] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:36.960461 22144 ts_tablet_manager.cc:1397] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.125s user 0.002s sys 0.005s
I20250112 02:57:36.962603 22144 raft_consensus.cc:357] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.963264 22144 raft_consensus.cc:383] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:36.963480 22144 raft_consensus.cc:738] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:36.964058 22144 consensus_queue.cc:260] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.967484 22144 ts_tablet_manager.cc:1428] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250112 02:57:36.968586 22145 tablet_bootstrap.cc:654] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:36.981501 23184 ts_tablet_manager.cc:1918] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:57:36.986665 23184 log.cc:1198] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/a4b543ed18b34cd58a1c588ed7d079c5
I20250112 02:57:36.987380 23184 ts_tablet_manager.cc:1939] T a4b543ed18b34cd58a1c588ed7d079c5 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:36.991187 23169 raft_consensus.cc:491] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.991655 23169 raft_consensus.cc:513] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.993151 23215 raft_consensus.cc:491] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.993515 23215 raft_consensus.cc:513] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.995280 23215 leader_election.cc:290] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:36.995764 23215 raft_consensus.cc:491] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:36.996186 23215 raft_consensus.cc:513] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.997861 23215 leader_election.cc:290] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:36.993518 23169 leader_election.cc:290] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.004561 23169 raft_consensus.cc:491] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.005019 23169 raft_consensus.cc:513] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:36.944290 22143 tablet_bootstrap.cc:654] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.009203 21753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.010460 21978 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.010994 23169 leader_election.cc:290] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.013859 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.014940 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.015918 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08141e3615244a9b266cd197d9b9ef3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:37.016255 21691 consensus_peers.cc:487] T f3da552a973945e284a386304f182777 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.016712 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9f8cbfcd4c41e2bdf8c04216a318bf" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:37.017788 21980 leader_election.cc:343] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:37.018437 21980 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.018848 21980 leader_election.cc:304] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.003062 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9f8cbfcd4c41e2bdf8c04216a318bf" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.019938 21980 leader_election.cc:343] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.003436 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08141e3615244a9b266cd197d9b9ef3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:37.004148 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.020484 21980 leader_election.cc:343] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:37.021350 21978 leader_election.cc:343] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.021708 21978 leader_election.cc:304] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.021922 23215 raft_consensus.cc:2743] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.022380 23215 raft_consensus.cc:2743] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:37.025224 21978 leader_election.cc:343] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.025588 21978 leader_election.cc:304] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.026160 23169 raft_consensus.cc:2743] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:37.027878 21978 leader_election.cc:343] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.028383 21978 leader_election.cc:304] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.029321 23215 raft_consensus.cc:2743] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.029811 22144 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:37.032400 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet f445a4bf43db4571b6d3a4afe8fe823a (table default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]) successfully deleted
I20250112 02:57:37.036592 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 280) took 1882 ms. Trace:
I20250112 02:57:37.036973 23186 rpcz_store.cc:276] 0112 02:57:35.154635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.154884 (+ 249us) service_pool.cc:225] Handling call
0112 02:57:37.034349 (+1879465us) ts_tablet_manager.cc:1145] Deleting tablet 15c16031f5b74cf9885fac110f445e1a
0112 02:57:37.034415 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:37.036528 (+ 2113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860793}
W20250112 02:57:37.049254 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 15c16031f5b74cf9885fac110f445e1a because the tablet was not found. No further retry: Not found: Tablet not found: 15c16031f5b74cf9885fac110f445e1a
I20250112 02:57:37.055485 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 281) took 1899 ms. Trace:
I20250112 02:57:37.055956 23186 rpcz_store.cc:276] 0112 02:57:35.155692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.155783 (+ 91us) service_pool.cc:225] Handling call
0112 02:57:37.055114 (+1899331us) ts_tablet_manager.cc:1145] Deleting tablet 4e2b248224f9416faf4bd83a853ec918
0112 02:57:37.055175 (+ 61us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:37.055422 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880109}
I20250112 02:57:37.063186 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet a4b543ed18b34cd58a1c588ed7d079c5 (table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]) successfully deleted
W20250112 02:57:37.064795 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 03d2021384bb41b58a66fcb7afaeaca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d2021384bb41b58a66fcb7afaeaca8 already in progress: creating tablet
I20250112 02:57:37.064488 23267 tablet.cc:1721] T 65eb8c99917247edbdbf5de42f3b505c P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:37.066157 21378 catalog_manager.cc:4852] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 247f0ef0ada44d9d957c8ff42b0da4b5 already in progress: creating tablet
I20250112 02:57:37.076588 21899 raft_consensus.cc:3054] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.085937 23194 consensus_queue.cc:1035] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.113034 23235 raft_consensus.cc:491] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.114419 23235 raft_consensus.cc:513] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.116528 22144 tablet_bootstrap.cc:654] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.134306 23235 leader_election.cc:290] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.134704 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779badaa22e6404aaaf1818e86228bd4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:37.135819 21980 leader_election.cc:343] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.136399 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779badaa22e6404aaaf1818e86228bd4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.137719 21978 leader_election.cc:343] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.138183 21978 leader_election.cc:304] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.138978 23215 raft_consensus.cc:2743] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.144438 22143 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:37.144802 22143 ts_tablet_manager.cc:1397] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.280s user 0.010s sys 0.007s
I20250112 02:57:37.144937 23192 raft_consensus.cc:491] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.145850 23192 raft_consensus.cc:513] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.148285 23192 leader_election.cc:290] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.123148 23273 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:37.151005 23273 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:37.154826 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:37.156095 21691 leader_election.cc:343] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:37.156777 21981 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 87839.6us
I20250112 02:57:37.159205 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:37.159976 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 4 candidate_status { last_received { term: 3 index: 306 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:37.161176 22841 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:37.177192 22145 tablet_bootstrap.cc:492] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:37.177702 22145 ts_tablet_manager.cc:1397] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.255s user 0.023s sys 0.030s
I20250112 02:57:37.178712 23273 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:37.179544 23273 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:57:37.180177 22145 raft_consensus.cc:357] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.180781 22145 raft_consensus.cc:383] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.180999 22145 raft_consensus.cc:738] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.181468 22145 consensus_queue.cc:260] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.182825 22145 ts_tablet_manager.cc:1428] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250112 02:57:37.189898 22144 tablet_bootstrap.cc:492] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:37.190316 22144 ts_tablet_manager.cc:1397] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.161s user 0.010s sys 0.000s
I20250112 02:57:37.192554 22144 raft_consensus.cc:357] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.193150 22144 raft_consensus.cc:383] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.193455 22144 raft_consensus.cc:738] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.202347 23186 tablet_replica.cc:331] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:37.203315 23186 raft_consensus.cc:2238] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:37.204257 23186 raft_consensus.cc:2267] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:37.194060 22144 consensus_queue.cc:260] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.221158 22144 ts_tablet_manager.cc:1428] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250112 02:57:37.221921 22144 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:37.222563 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:37.223133 22040 raft_consensus.cc:2463] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:37.229552 22143 raft_consensus.cc:357] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.230213 22143 raft_consensus.cc:383] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.230456 22143 raft_consensus.cc:738] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
W20250112 02:57:37.057209 21376 catalog_manager.cc:4839] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 4e2b248224f9416faf4bd83a853ec918 because the tablet was not found. No further retry: Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
W20250112 02:57:37.233721 21376 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 176618us
I20250112 02:57:37.235248 22092 heartbeater.cc:346] Connected to a master server at 127.20.217.190:34893
I20250112 02:57:37.237401 22143 consensus_queue.cc:260] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.244045 22143 ts_tablet_manager.cc:1428] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.099s user 0.006s sys 0.000s
W20250112 02:57:37.254947 22841 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.188:35683, user_credentials={real_user=slave}} blocked reactor thread for 93886.7us
I20250112 02:57:37.285863 22145 tablet_bootstrap.cc:492] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:37.291368 22145 tablet_bootstrap.cc:654] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.302111 23192 consensus_queue.cc:1035] T a69123c4ab384e0380c62e52b9f3b42d P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.311702 21690 leader_election.cc:304] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:37.311988 21691 consensus_peers.cc:487] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.312723 23022 raft_consensus.cc:2798] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:37.313009 23022 raft_consensus.cc:491] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:37.313262 23022 raft_consensus.cc:3054] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.316704 22144 tablet_bootstrap.cc:654] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.320266 23022 raft_consensus.cc:513] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.322173 23022 leader_election.cc:290] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.324826 23124 consensus_queue.cc:1035] T 4716677cfad74c0dafe18c55796aeae5 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.327659 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:37.330992 21691 leader_election.cc:343] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.206452 23186 ts_tablet_manager.cc:1905] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:37.336825 22143 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:37.336977 23268 raft_consensus.cc:491] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.337386 23268 raft_consensus.cc:513] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.339191 23268 leader_election.cc:290] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.339344 22144 tablet_bootstrap.cc:492] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:37.339738 22144 ts_tablet_manager.cc:1397] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.000s
I20250112 02:57:37.340525 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfa495bfefc46b998c2f870d9014900" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:37.341639 21980 leader_election.cc:343] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.341686 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfa495bfefc46b998c2f870d9014900" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:37.341851 22144 raft_consensus.cc:357] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:37.343572 21978 leader_election.cc:343] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.343935 21978 leader_election.cc:304] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.345058 23235 raft_consensus.cc:2743] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.352337 22145 tablet_bootstrap.cc:492] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:37.352916 22145 ts_tablet_manager.cc:1397] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
W20250112 02:57:37.362399 21691 consensus_peers.cc:487] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.363965 22143 tablet_bootstrap.cc:654] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.363842 22145 raft_consensus.cc:357] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.364758 22145 raft_consensus.cc:383] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.365070 22145 raft_consensus.cc:738] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.366714 22145 consensus_queue.cc:260] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.369807 22145 ts_tablet_manager.cc:1428] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20250112 02:57:37.370708 22145 tablet_bootstrap.cc:492] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:37.342471 22144 raft_consensus.cc:383] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.373932 22144 raft_consensus.cc:738] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.377260 22145 tablet_bootstrap.cc:654] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.377308 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:37.377847 22044 raft_consensus.cc:3054] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.383708 22044 raft_consensus.cc:2463] T 58c49391d2ed4c3fa84fa2d071b06094 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:37.388839 22144 consensus_queue.cc:260] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.391139 21690 leader_election.cc:304] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.392084 23022 raft_consensus.cc:2798] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:37.392558 23022 raft_consensus.cc:695] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:37.393384 23022 consensus_queue.cc:237] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.407863 22144 ts_tablet_manager.cc:1428] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.068s user 0.005s sys 0.002s
I20250112 02:57:37.424799 22144 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:37.448611 23246 consensus_queue.cc:1035] T 66fe67a814504a89b6fa8f46e2a4aee9 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.444257 23124 raft_consensus.cc:491] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.450050 23124 raft_consensus.cc:513] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.451879 23124 leader_election.cc:290] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.452522 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.457325 22144 tablet_bootstrap.cc:654] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:37.458141 21691 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.482442 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:37.483110 22044 raft_consensus.cc:2463] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:37.488534 23022 consensus_queue.cc:1035] T ae3ed0bdc2ec44998ee461d4b1254e91 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.490164 21690 leader_election.cc:304] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.491034 23022 raft_consensus.cc:2798] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:37.491324 23022 raft_consensus.cc:491] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:37.491596 23022 raft_consensus.cc:3054] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.511415 22143 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:37.512039 22143 ts_tablet_manager.cc:1397] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.175s user 0.010s sys 0.000s
I20250112 02:57:37.513340 23022 raft_consensus.cc:513] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.520166 22143 raft_consensus.cc:357] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.542923 22145 tablet_bootstrap.cc:492] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:37.543298 22145 ts_tablet_manager.cc:1397] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.173s user 0.020s sys 0.011s
I20250112 02:57:37.549012 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:37.545938 22145 raft_consensus.cc:357] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.552407 22145 raft_consensus.cc:383] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.552742 22145 raft_consensus.cc:738] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.553416 22145 consensus_queue.cc:260] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.558102 22145 ts_tablet_manager.cc:1428] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250112 02:57:37.558900 22145 tablet_bootstrap.cc:492] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:37.521124 23192 raft_consensus.cc:491] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.561131 22143 raft_consensus.cc:383] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.566179 22143 raft_consensus.cc:738] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.567783 22143 consensus_queue.cc:260] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.570009 22145 tablet_bootstrap.cc:654] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.573845 22015 tablet_service.cc:1467] Processing CreateTablet for tablet f226759c7d8b4b0d83395c542428f4c9 (DEFAULT_TABLE table=default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.580327 22145 tablet_bootstrap.cc:492] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:37.580166 21872 tablet_service.cc:1467] Processing CreateTablet for tablet f226759c7d8b4b0d83395c542428f4c9 (DEFAULT_TABLE table=default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.580631 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f226759c7d8b4b0d83395c542428f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.580706 22145 ts_tablet_manager.cc:1397] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.009s
I20250112 02:57:37.581071 21872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f226759c7d8b4b0d83395c542428f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.582836 22145 raft_consensus.cc:357] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.583448 22145 raft_consensus.cc:383] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.583688 22145 raft_consensus.cc:738] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.584347 21882 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4262c2fdc0440ea696e551d00b80e9 (DEFAULT_TABLE table=default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.584324 22145 consensus_queue.cc:260] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.585240 21882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4262c2fdc0440ea696e551d00b80e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.587695 23242 raft_consensus.cc:491] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.588089 23242 raft_consensus.cc:513] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:37.588995 21691 leader_election.cc:343] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.589830 23242 leader_election.cc:290] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.590227 23022 leader_election.cc:290] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.596812 22145 ts_tablet_manager.cc:1428] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250112 02:57:37.597723 22145 tablet_bootstrap.cc:492] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:37.596874 22144 tablet_bootstrap.cc:492] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:37.601210 22144 ts_tablet_manager.cc:1397] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.177s user 0.010s sys 0.000s
I20250112 02:57:37.604663 22144 raft_consensus.cc:357] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.605443 22144 raft_consensus.cc:383] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.605758 22144 raft_consensus.cc:738] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.606452 22144 consensus_queue.cc:260] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.609813 22008 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4262c2fdc0440ea696e551d00b80e9 (DEFAULT_TABLE table=default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.610752 22008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4262c2fdc0440ea696e551d00b80e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.625140 22014 tablet_service.cc:1467] Processing CreateTablet for tablet 8b39a5b05b0941aea541a250bb64941e (DEFAULT_TABLE table=default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.626112 22014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b39a5b05b0941aea541a250bb64941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.539551 23186 ts_tablet_manager.cc:1918] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:37.626817 23186 log.cc:1198] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:37.595172 23192 raft_consensus.cc:513] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.629005 23192 leader_election.cc:290] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.631256 21727 tablet_service.cc:1467] Processing CreateTablet for tablet f226759c7d8b4b0d83395c542428f4c9 (DEFAULT_TABLE table=default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.632135 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f226759c7d8b4b0d83395c542428f4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.627609 23186 ts_tablet_manager.cc:1939] T 2601daad8ba742ec88ccfe603440e8e7 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:37.636435 22019 tablet_service.cc:1467] Processing CreateTablet for tablet e0e6beaabb6d4b099ce9e47736bb29b4 (DEFAULT_TABLE table=default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.637451 22019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e6beaabb6d4b099ce9e47736bb29b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.643100 23242 raft_consensus.cc:491] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.644730 23242 raft_consensus.cc:513] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.647141 22144 ts_tablet_manager.cc:1428] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.045s user 0.006s sys 0.000s
I20250112 02:57:37.647701 23242 leader_election.cc:290] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.658780 22007 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef7ac20f034f589e564e5f5e754705 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.659723 22007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7ac20f034f589e564e5f5e754705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.663744 22144 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:37.669409 22040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:37.670387 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef7ac20f034f589e564e5f5e754705 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.670928 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 8b39a5b05b0941aea541a250bb64941e (DEFAULT_TABLE table=default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.671319 21877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7ac20f034f589e564e5f5e754705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.671916 21880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b39a5b05b0941aea541a250bb64941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.672581 21874 tablet_service.cc:1467] Processing CreateTablet for tablet 05ba915163894cc6aa0cfb144dd49d97 (DEFAULT_TABLE table=default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.672986 21876 tablet_service.cc:1467] Processing CreateTablet for tablet 95f62433f15a43eb9a654c067052cd68 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.673377 21874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba915163894cc6aa0cfb144dd49d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.673847 21876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62433f15a43eb9a654c067052cd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.675177 21879 tablet_service.cc:1467] Processing CreateTablet for tablet e0e6beaabb6d4b099ce9e47736bb29b4 (DEFAULT_TABLE table=default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.676038 21879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e6beaabb6d4b099ce9e47736bb29b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.685691 22012 tablet_service.cc:1467] Processing CreateTablet for tablet 05ba915163894cc6aa0cfb144dd49d97 (DEFAULT_TABLE table=default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.686612 22012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba915163894cc6aa0cfb144dd49d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.687867 22145 tablet_bootstrap.cc:654] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.694754 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]). Expected version 1 got 0
I20250112 02:57:37.695353 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:37.696334 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]). Expected version 1 got 0
I20250112 02:57:37.697614 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:37.697904 22144 tablet_bootstrap.cc:654] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:37.709425 21836 leader_election.cc:343] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 5bd0d70fb0fe4a179b1ea6fa23d180c9
I20250112 02:57:37.710058 23242 raft_consensus.cc:491] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.710614 23242 raft_consensus.cc:513] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.713158 23242 leader_election.cc:290] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.717722 22015 tablet_service.cc:1467] Processing CreateTablet for tablet 95f62433f15a43eb9a654c067052cd68 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.720847 22143 ts_tablet_manager.cc:1428] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.209s user 0.004s sys 0.001s
I20250112 02:57:37.722929 21755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:37.724510 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 4e4262c2fdc0440ea696e551d00b80e9 (DEFAULT_TABLE table=default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.725239 21730 tablet_service.cc:1467] Processing CreateTablet for tablet 8b39a5b05b0941aea541a250bb64941e (DEFAULT_TABLE table=default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.732278 21736 tablet_service.cc:1467] Processing CreateTablet for tablet e0e6beaabb6d4b099ce9e47736bb29b4 (DEFAULT_TABLE table=default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.733500 21726 tablet_service.cc:1467] Processing CreateTablet for tablet 3fef7ac20f034f589e564e5f5e754705 (DEFAULT_TABLE table=default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.734398 21896 raft_consensus.cc:3054] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.734962 21725 tablet_service.cc:1467] Processing CreateTablet for tablet 95f62433f15a43eb9a654c067052cd68 (DEFAULT_TABLE table=default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.735113 21731 tablet_service.cc:1467] Processing CreateTablet for tablet 05ba915163894cc6aa0cfb144dd49d97 (DEFAULT_TABLE table=default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:57:37.736344 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.741446 21754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bd0d70fb0fe4a179b1ea6fa23d180c9" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:37.747233 21835 leader_election.cc:304] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:37.748216 23297 raft_consensus.cc:2743] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.748566 21835 leader_election.cc:304] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:37.747998 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:37.750358 21751 raft_consensus.cc:2463] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:37.752123 21835 leader_election.cc:304] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:37.758956 22143 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:37.759805 21726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fef7ac20f034f589e564e5f5e754705. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.763002 21725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62433f15a43eb9a654c067052cd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.765609 22143 tablet_bootstrap.cc:654] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:37.767441 21736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0e6beaabb6d4b099ce9e47736bb29b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.768854 23305 raft_consensus.cc:2798] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:37.769160 23305 raft_consensus.cc:491] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:37.769431 23305 raft_consensus.cc:3054] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:37.771049 21980 consensus_peers.cc:487] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.771464 23297 raft_consensus.cc:2743] T b0d0ba05ee4c4c9f815a4c04830b5dd4 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.774135 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e4262c2fdc0440ea696e551d00b80e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:57:37.779686 21691 leader_election.cc:343] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:37.780262 21691 consensus_peers.cc:487] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.786926 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 282) took 2630 ms. Trace:
I20250112 02:57:37.787479 23186 rpcz_store.cc:276] 0112 02:57:35.156430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.156531 (+ 101us) service_pool.cc:225] Handling call
0112 02:57:37.163269 (+2006738us) ts_tablet_manager.cc:1145] Deleting tablet 2601daad8ba742ec88ccfe603440e8e7
0112 02:57:37.202196 (+ 38927us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:37.513351 (+311155us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:37.539265 (+ 25914us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:37.755667 (+216402us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b5800001898. stack: 00007fe304724853 00007fe304724750 00007fe304299584 00000000004fcd24 00000000004fccb9 00007fe30459a2d1 00007fe304719291 00007fe3047198c2 00007fe30d3ab999 00007fe30c6105f8 00007fe30c6d63de 00007fe30c6d9289 00007fe30c70144e 00007fe30c701419 00007fe30c7013d5 00007fe30c6f3146
0112 02:57:37.786858 (+ 31191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13801856,"tablet-delete.queue_time_us":1988123}
I20250112 02:57:37.789796 23186 tablet_replica.cc:331] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: stopping tablet replica
I20250112 02:57:37.801601 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet 2601daad8ba742ec88ccfe603440e8e7 (table default.Table_33ea8d4190104eb8837824b516dd2314 [id=097357eb9ce548aa9fe2db1a906c70d8]) successfully deleted
I20250112 02:57:37.789044 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ba915163894cc6aa0cfb144dd49d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.818063 23268 raft_consensus.cc:491] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.818495 23268 raft_consensus.cc:513] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.821164 23192 consensus_queue.cc:1035] T be5f2a4f6a8e4a64a7642d5a18b58a6d P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:37.823051 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b39a5b05b0941aea541a250bb64941e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.823599 22039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:37.824157 22039 raft_consensus.cc:2463] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:37.825255 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d017c05abe94657bfbd2b38329bba16" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:37.825814 22038 raft_consensus.cc:3054] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:37.825836 21378 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet eab717cb5d794d15b9b51881b86501fc (table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet eab717cb5d794d15b9b51881b86501fc consensus state has no leader
I20250112 02:57:37.827611 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
W20250112 02:57:37.830005 21836 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: a2ce633c5b3b4cbea17c171564fa72d6
I20250112 02:57:37.839184 21690 leader_election.cc:304] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.839506 23245 consensus_queue.cc:1035] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:37.840420 23124 raft_consensus.cc:2798] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:37.840775 23124 raft_consensus.cc:491] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:37.841125 23124 raft_consensus.cc:3054] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.850984 23305 raft_consensus.cc:513] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.853212 23305 leader_election.cc:290] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.853997 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:37.854722 21757 raft_consensus.cc:3054] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:37.855726 21836 leader_election.cc:343] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: a2ce633c5b3b4cbea17c171564fa72d6
I20250112 02:57:37.856184 23268 leader_election.cc:290] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.860540 21757 raft_consensus.cc:2463] T a2ce633c5b3b4cbea17c171564fa72d6 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:37.861465 21835 leader_election.cc:304] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters: e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:37.834278 22038 raft_consensus.cc:2463] T 5d017c05abe94657bfbd2b38329bba16 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:37.866261 22144 tablet_bootstrap.cc:492] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:37.861989 23124 raft_consensus.cc:513] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.869014 23124 leader_election.cc:290] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:37.871532 22145 tablet_bootstrap.cc:492] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:37.867445 21690 leader_election.cc:304] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.873627 23298 raft_consensus.cc:2798] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:37.874204 23298 raft_consensus.cc:695] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:37.875133 23298 consensus_queue.cc:237] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:37.877482 21691 consensus_peers.cc:487] T 78f03a1dfb604b33a0528c28ade287f2 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:37.879374 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:37.879976 22038 raft_consensus.cc:3054] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:37.885835 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e25cf536254f3ea9d2c1e97a32b85b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.887063 21978 leader_election.cc:343] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.829314 21392 catalog_manager.cc:5618] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:37.893014 22038 raft_consensus.cc:2463] T 13edf2c56e8e4254b75e0eb85fb88427 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:37.895725 22144 ts_tablet_manager.cc:1397] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.232s user 0.008s sys 0.004s
I20250112 02:57:37.874308 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e25cf536254f3ea9d2c1e97a32b85b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.876569 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:37.872012 23305 raft_consensus.cc:2798] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:37.896826 23305 raft_consensus.cc:695] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:37.897621 23305 consensus_queue.cc:237] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:37.897544 21980 leader_election.cc:343] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.899650 21980 leader_election.cc:304] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.900661 23307 raft_consensus.cc:2743] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.899112 22145 ts_tablet_manager.cc:1397] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.302s user 0.011s sys 0.004s
I20250112 02:57:37.903592 23307 raft_consensus.cc:491] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:37.903898 21690 leader_election.cc:304] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:37.904006 23307 raft_consensus.cc:513] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.905391 23022 raft_consensus.cc:2798] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:37.905849 23022 raft_consensus.cc:695] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:37.906203 23307 leader_election.cc:290] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:37.906577 23022 consensus_queue.cc:237] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.908317 22015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f62433f15a43eb9a654c067052cd68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:37.908646 22145 raft_consensus.cc:357] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.909217 22145 raft_consensus.cc:383] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.909489 22145 raft_consensus.cc:738] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.916319 22145 consensus_queue.cc:260] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.919644 22145 ts_tablet_manager.cc:1428] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
W20250112 02:57:37.919901 21691 leader_election.cc:343] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.920367 22145 tablet_bootstrap.cc:492] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:37.920923 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94cda34ff2e44cfa543362a007f1400" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:37.922362 21978 leader_election.cc:343] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.942391 23186 raft_consensus.cc:2238] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:57:37.943259 23186 raft_consensus.cc:2267] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:37.945214 23186 ts_tablet_manager.cc:1905] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:37.906618 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d94cda34ff2e44cfa543362a007f1400" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:37.919890 21900 raft_consensus.cc:3054] T 5bd0d70fb0fe4a179b1ea6fa23d180c9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:37.958995 21980 leader_election.cc:343] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:37.959370 21980 leader_election.cc:304] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:37.961145 23308 raft_consensus.cc:2743] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:37.903326 22144 raft_consensus.cc:357] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.983817 22144 raft_consensus.cc:383] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:37.984114 22144 raft_consensus.cc:738] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:37.984867 22144 consensus_queue.cc:260] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:37.992374 22144 ts_tablet_manager.cc:1428] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.091s user 0.005s sys 0.001s
I20250112 02:57:38.023383 22144 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:38.035717 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 76781.9us
I20250112 02:57:38.039455 22145 tablet_bootstrap.cc:654] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.040885 23308 raft_consensus.cc:491] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.041253 23308 raft_consensus.cc:513] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.043077 23308 leader_election.cc:290] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:38.051517 21751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2021384bb41b58a66fcb7afaeaca8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:38.053287 21978 leader_election.cc:343] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.084991 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2021384bb41b58a66fcb7afaeaca8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.087472 22144 tablet_bootstrap.cc:654] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:38.091351 21980 leader_election.cc:343] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.091542 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:38.091801 21980 leader_election.cc:304] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.092450 23308 raft_consensus.cc:2743] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:38.092630 21391 catalog_manager.cc:5526] T a69123c4ab384e0380c62e52b9f3b42d P cca160b7cffb4ae287ea077e7352ffc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129). New cstate: current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } }
I20250112 02:57:38.093670 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:38.096689 23321 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:38.097326 23321 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.099167 23186 ts_tablet_manager.cc:1918] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:38.099578 23186 log.cc:1198] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/a4b543ed18b34cd58a1c588ed7d079c5
I20250112 02:57:38.105194 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 365 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:38.108999 23321 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:38.120762 23308 consensus_queue.cc:1035] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.117743 23186 ts_tablet_manager.cc:1939] T a4b543ed18b34cd58a1c588ed7d079c5 P 1c2770701ca54288956df9d910d7d3d2: Deleting consensus metadata
I20250112 02:57:38.138885 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 285) took 1989 ms. Trace:
I20250112 02:57:38.139240 23186 rpcz_store.cc:276] 0112 02:57:36.149811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.150183 (+ 372us) service_pool.cc:225] Handling call
0112 02:57:37.789635 (+1639452us) ts_tablet_manager.cc:1145] Deleting tablet a4b543ed18b34cd58a1c588ed7d079c5
0112 02:57:37.789685 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:37.998607 (+208922us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:38.098937 (+100330us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:38.138826 (+ 39889us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":119022,"spinlock_wait_cycles":2304,"tablet-delete.queue_time_us":1579233}
I20250112 02:57:38.140766 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 286) took 1706 ms. Trace:
I20250112 02:57:38.141110 23186 rpcz_store.cc:276] 0112 02:57:36.441634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.441907 (+ 273us) service_pool.cc:225] Handling call
0112 02:57:38.140332 (+1698425us) ts_tablet_manager.cc:1145] Deleting tablet 03d2021384bb41b58a66fcb7afaeaca8
0112 02:57:38.140387 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:38.140704 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697380}
I20250112 02:57:38.142230 23186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46754 (request call id 288) took 1294 ms. Trace:
I20250112 02:57:38.146826 21376 catalog_manager.cc:4872] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): tablet a4b543ed18b34cd58a1c588ed7d079c5 (table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]) successfully deleted
W20250112 02:57:38.147529 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 03d2021384bb41b58a66fcb7afaeaca8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03d2021384bb41b58a66fcb7afaeaca8 already in progress: creating tablet
I20250112 02:57:38.148824 23186 rpcz_store.cc:276] 0112 02:57:36.847670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.852911 (+ 5241us) service_pool.cc:225] Handling call
0112 02:57:38.141909 (+1288998us) ts_tablet_manager.cc:1145] Deleting tablet 247f0ef0ada44d9d957c8ff42b0da4b5
0112 02:57:38.141945 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:38.142185 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285478}
W20250112 02:57:38.149870 21376 catalog_manager.cc:4852] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): delete failed for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 247f0ef0ada44d9d957c8ff42b0da4b5 already in progress: creating tablet
I20250112 02:57:38.094287 21391 catalog_manager.cc:5618] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:38.150465 22143 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:38.150873 22143 ts_tablet_manager.cc:1397] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.392s user 0.013s sys 0.054s
I20250112 02:57:38.163232 22145 tablet_bootstrap.cc:492] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.163619 22145 ts_tablet_manager.cc:1397] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.243s user 0.007s sys 0.004s
I20250112 02:57:38.177083 22143 raft_consensus.cc:357] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.177742 22143 raft_consensus.cc:383] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.177982 22143 raft_consensus.cc:738] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.178581 22143 consensus_queue.cc:260] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.183599 22144 tablet_bootstrap.cc:492] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:38.183965 22144 ts_tablet_manager.cc:1397] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.161s user 0.006s sys 0.003s
I20250112 02:57:38.186653 22144 raft_consensus.cc:357] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.187361 22144 raft_consensus.cc:383] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.187624 22144 raft_consensus.cc:738] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.188236 22144 consensus_queue.cc:260] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.190312 22144 ts_tablet_manager.cc:1428] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 02:57:38.191869 22143 ts_tablet_manager.cc:1428] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20250112 02:57:38.192703 22143 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:38.209627 22145 raft_consensus.cc:357] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.210283 22145 raft_consensus.cc:383] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.210532 22145 raft_consensus.cc:738] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.216498 23299 raft_consensus.cc:491] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.216521 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 365 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:38.216924 23299 raft_consensus.cc:513] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.217283 22881 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 3.
I20250112 02:57:38.218626 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd
I20250112 02:57:38.218801 23299 leader_election.cc:290] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.219219 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.219904 23321 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250112 02:57:38.220186 23321 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcc0757376c3411a9fcbc84ec411befd)
I20250112 02:57:38.220502 23321 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:57:38.220506 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:38.220742 22144 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:38.220878 21691 leader_election.cc:343] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.221046 22038 raft_consensus.cc:2463] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:38.222012 21690 leader_election.cc:304] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.223502 23192 raft_consensus.cc:2798] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:38.223834 23192 raft_consensus.cc:491] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:38.224133 23192 raft_consensus.cc:3054] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.228111 23321 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.229566 22143 tablet_bootstrap.cc:654] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.234712 23192 raft_consensus.cc:513] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.248745 23192 leader_election.cc:290] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.250303 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:38.250792 22038 raft_consensus.cc:3054] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.255040 21554 raft_consensus.cc:1235] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 FOLLOWER]: Rejecting Update request from peer bcc0757376c3411a9fcbc84ec411befd for earlier term 3. Current term is 4. Ops: [3.366-3.366]
I20250112 02:57:38.256685 22038 raft_consensus.cc:2463] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:38.256976 23293 consensus_queue.cc:1046] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 }, Status: INVALID_TERM, Last received: 3.365, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20250112 02:57:38.257649 21690 leader_election.cc:304] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:38.260264 23299 raft_consensus.cc:2798] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:38.260774 23299 raft_consensus.cc:695] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:38.261459 22145 consensus_queue.cc:260] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.261773 23299 consensus_queue.cc:237] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.263589 22145 ts_tablet_manager.cc:1428] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.100s user 0.006s sys 0.000s
I20250112 02:57:38.265542 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 365 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd"
I20250112 02:57:38.268671 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 4 candidate_status { last_received { term: 3 index: 365 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:38.269178 22881 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:57:38.226356 22144 tablet_bootstrap.cc:654] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.249665 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:38.270371 21691 leader_election.cc:343] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.275384 22145 tablet_bootstrap.cc:492] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.275657 22143 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:38.276085 22143 ts_tablet_manager.cc:1397] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.003s
I20250112 02:57:38.278705 22143 raft_consensus.cc:357] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.279524 22881 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 4.
I20250112 02:57:38.280479 21533 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd
I20250112 02:57:38.290688 23321 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 4 election: Requested vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:38.291338 23321 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 FOLLOWER]: Leader election won for term 4
I20250112 02:57:38.291785 23321 raft_consensus.cc:695] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 LEADER]: Becoming Leader. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Running, Role: LEADER
I20250112 02:57:38.292680 23321 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 3.365, Last appended by leader: 365, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.296838 22145 tablet_bootstrap.cc:654] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.299926 22144 tablet_bootstrap.cc:492] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:38.300376 22144 ts_tablet_manager.cc:1397] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.003s
I20250112 02:57:38.302704 22144 raft_consensus.cc:357] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.308424 22143 raft_consensus.cc:383] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.308720 22143 raft_consensus.cc:738] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.309386 22143 consensus_queue.cc:260] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.311861 22143 ts_tablet_manager.cc:1428] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.035s user 0.005s sys 0.000s
I20250112 02:57:38.312177 22144 raft_consensus.cc:383] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.312489 22144 raft_consensus.cc:738] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.313180 22144 consensus_queue.cc:260] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.314661 22143 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:38.318974 23329 raft_consensus.cc:3049] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 LEADER]: Stepping down as leader of term 3
I20250112 02:57:38.319319 23329 raft_consensus.cc:738] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcc0757376c3411a9fcbc84ec411befd, State: Running, Role: LEADER
I20250112 02:57:38.321020 22144 ts_tablet_manager.cc:1428] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20250112 02:57:38.325973 22145 tablet_bootstrap.cc:492] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.326352 22145 ts_tablet_manager.cc:1397] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.000s
I20250112 02:57:38.333642 23329 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 3.366, Last appended by leader: 366, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.335541 23329 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 3 FOLLOWER]: Advancing to term 4
I20250112 02:57:38.336514 22144 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:38.339185 22145 raft_consensus.cc:357] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.339824 22145 raft_consensus.cc:383] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.340490 23299 raft_consensus.cc:491] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.340898 23299 raft_consensus.cc:513] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.342622 23299 leader_election.cc:290] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.344022 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.344241 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:38.344568 21900 raft_consensus.cc:2463] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:38.344743 22038 raft_consensus.cc:2463] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:38.344982 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 4 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:38.345698 21690 leader_election.cc:304] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:38.345925 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:38.340065 22145 raft_consensus.cc:738] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.346561 23299 raft_consensus.cc:2798] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:38.346894 23299 raft_consensus.cc:491] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:38.347211 23299 raft_consensus.cc:3054] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.354916 23308 raft_consensus.cc:491] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.355327 23308 raft_consensus.cc:513] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.354605 23299 raft_consensus.cc:513] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.365288 23334 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:57:38.367115 22145 consensus_queue.cc:260] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.368918 23308 leader_election.cc:290] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:38.369599 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.369782 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:38.370718 21980 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:38.371166 21978 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.371537 21978 leader_election.cc:304] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.373461 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:38.373914 21900 raft_consensus.cc:3054] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.378813 23308 raft_consensus.cc:2743] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:38.380569 22145 ts_tablet_manager.cc:1428] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250112 02:57:38.381753 21900 raft_consensus.cc:2463] T 2d793344526a404e8a737e7fb628e521 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:38.382766 21691 leader_election.cc:304] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:38.383620 23124 raft_consensus.cc:2798] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:38.384030 23124 raft_consensus.cc:695] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:38.384763 23124 consensus_queue.cc:237] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.390760 22143 tablet_bootstrap.cc:654] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.394731 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d793344526a404e8a737e7fb628e521" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:38.395234 22045 raft_consensus.cc:3054] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.395906 22145 tablet_bootstrap.cc:492] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.391475 23299 leader_election.cc:290] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.400914 22045 raft_consensus.cc:2463] T 2d793344526a404e8a737e7fb628e521 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:38.405280 22145 tablet_bootstrap.cc:654] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.406375 22144 tablet_bootstrap.cc:654] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:38.426955 21691 consensus_peers.cc:487] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:38.433756 22144 tablet_bootstrap.cc:492] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:38.434387 22144 ts_tablet_manager.cc:1397] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.002s
I20250112 02:57:38.444468 22145 tablet_bootstrap.cc:492] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.444854 22145 ts_tablet_manager.cc:1397] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.006s
I20250112 02:57:38.447618 22145 raft_consensus.cc:357] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.448309 22145 raft_consensus.cc:383] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.448539 22145 raft_consensus.cc:738] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.451859 22145 consensus_queue.cc:260] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.452934 22143 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:38.453379 22143 ts_tablet_manager.cc:1397] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.139s user 0.009s sys 0.002s
I20250112 02:57:38.455596 22143 raft_consensus.cc:357] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.459141 22144 raft_consensus.cc:357] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.459781 22143 raft_consensus.cc:383] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.464534 22143 raft_consensus.cc:738] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.465225 22143 consensus_queue.cc:260] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.466078 22144 raft_consensus.cc:383] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.466765 22144 raft_consensus.cc:738] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.468493 22145 ts_tablet_manager.cc:1428] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20250112 02:57:38.469118 22144 consensus_queue.cc:260] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.475852 22145 tablet_bootstrap.cc:492] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.484862 22145 tablet_bootstrap.cc:654] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.493908 23124 consensus_queue.cc:1035] T 58c49391d2ed4c3fa84fa2d071b06094 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.512840 22145 tablet_bootstrap.cc:492] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.513216 22145 ts_tablet_manager.cc:1397] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.000s
I20250112 02:57:38.514057 21899 raft_consensus.cc:3054] T 66fe67a814504a89b6fa8f46e2a4aee9 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.515645 22145 raft_consensus.cc:357] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.516247 22145 raft_consensus.cc:383] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.516461 22145 raft_consensus.cc:738] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.516973 22145 consensus_queue.cc:260] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.525347 22145 ts_tablet_manager.cc:1428] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250112 02:57:38.526134 22145 tablet_bootstrap.cc:492] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.540868 23308 raft_consensus.cc:491] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.541074 22145 tablet_bootstrap.cc:654] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.541277 23308 raft_consensus.cc:513] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.542987 23308 leader_election.cc:290] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:38.543391 22143 ts_tablet_manager.cc:1428] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250112 02:57:38.544593 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab717cb5d794d15b9b51881b86501fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:38.545805 21980 leader_election.cc:343] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.548472 22144 ts_tablet_manager.cc:1428] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.098s user 0.003s sys 0.002s
I20250112 02:57:38.557013 22144 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:38.562477 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab717cb5d794d15b9b51881b86501fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:38.563688 22143 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:38.563653 21978 leader_election.cc:343] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.563987 21978 leader_election.cc:304] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.564780 23308 raft_consensus.cc:2743] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:38.587389 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd]
I20250112 02:57:38.588999 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740]
I20250112 02:57:38.601248 22145 tablet_bootstrap.cc:492] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.601603 22145 ts_tablet_manager.cc:1397] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.022s
I20250112 02:57:38.603511 22145 raft_consensus.cc:357] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.604082 22145 raft_consensus.cc:383] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.604316 22145 raft_consensus.cc:738] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.604547 23308 raft_consensus.cc:491] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.604864 22145 consensus_queue.cc:260] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.604982 23308 raft_consensus.cc:513] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.607298 23308 leader_election.cc:290] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:38.612457 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae]
I20250112 02:57:38.612985 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e861bab8dcef41328095954a46e00a96" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:38.613749 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e]
W20250112 02:57:38.614085 21978 leader_election.cc:343] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.619792 22145 ts_tablet_manager.cc:1428] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 02:57:38.622486 22145 tablet_bootstrap.cc:492] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.608319 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e861bab8dcef41328095954a46e00a96" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:38.627339 21980 leader_election.cc:343] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.627697 21980 leader_election.cc:304] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.628523 23307 raft_consensus.cc:2743] T e861bab8dcef41328095954a46e00a96 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:38.621361 21899 raft_consensus.cc:3054] T 4716677cfad74c0dafe18c55796aeae5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.636483 22143 tablet_bootstrap.cc:654] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.651253 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]
I20250112 02:57:38.652647 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0]
I20250112 02:57:38.653870 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043]
I20250112 02:57:38.656092 23307 raft_consensus.cc:491] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.656577 23307 raft_consensus.cc:513] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.659857 22144 tablet_bootstrap.cc:654] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.668171 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c31714d18414800845671d783ac8173" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.668336 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c31714d18414800845671d783ac8173" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:38.669236 21980 leader_election.cc:343] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.669653 22145 tablet_bootstrap.cc:654] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:38.669777 21978 leader_election.cc:343] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.670107 21978 leader_election.cc:304] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.670751 23308 raft_consensus.cc:2743] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:38.671244 23307 leader_election.cc:290] T 9c31714d18414800845671d783ac8173 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:38.681311 22143 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:38.681792 22143 ts_tablet_manager.cc:1397] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.004s
I20250112 02:57:38.687453 22143 raft_consensus.cc:357] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.688051 22143 raft_consensus.cc:383] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.688279 22143 raft_consensus.cc:738] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.688784 22143 consensus_queue.cc:260] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.690666 22143 ts_tablet_manager.cc:1428] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250112 02:57:38.701692 22145 tablet_bootstrap.cc:492] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.702030 22145 ts_tablet_manager.cc:1397] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.004s
I20250112 02:57:38.707686 22145 raft_consensus.cc:357] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.708355 22145 raft_consensus.cc:383] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.708566 22145 raft_consensus.cc:738] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.709455 22144 tablet_bootstrap.cc:492] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:38.709216 22145 consensus_queue.cc:260] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.709801 22144 ts_tablet_manager.cc:1397] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.001s
I20250112 02:57:38.711158 22145 ts_tablet_manager.cc:1428] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250112 02:57:38.711833 22145 tablet_bootstrap.cc:492] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.712301 22144 raft_consensus.cc:357] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.712875 22144 raft_consensus.cc:383] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.713099 22144 raft_consensus.cc:738] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.715569 22144 consensus_queue.cc:260] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.716938 22881 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Refusing update from remote peer e76221f7d4ec44da987303d68c393a1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 306. Preceding OpId from leader: term: 4 index: 366. (index mismatch)
I20250112 02:57:38.717141 22145 tablet_bootstrap.cc:654] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.717916 22143 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:38.718650 22144 ts_tablet_manager.cc:1428] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250112 02:57:38.718382 23332 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 306, Time since last communication: 0.000s
I20250112 02:57:38.720782 22144 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:38.723397 23329 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.723783 23329 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.726059 23329 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:38.729050 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:38.729658 22878 raft_consensus.cc:2463] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcc0757376c3411a9fcbc84ec411befd in term 4.
I20250112 02:57:38.731392 21379 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e9ef739ef9dc460ca8287b1d2c6c2217; no voters:
I20250112 02:57:38.732157 23329 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250112 02:57:38.732538 23329 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:38.733065 23329 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 4 FOLLOWER]: Advancing to term 5
I20250112 02:57:38.733798 22144 tablet_bootstrap.cc:654] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.741225 22143 tablet_bootstrap.cc:654] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.745594 21900 raft_consensus.cc:3054] T 5d017c05abe94657bfbd2b38329bba16 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.743772 23329 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:38.751927 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:38.753963 23192 consensus_queue.cc:1035] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.756913 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748]
I20250112 02:57:38.756986 23329 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 election: Requested vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:38.794378 21902 raft_consensus.cc:3054] T 78f03a1dfb604b33a0528c28ade287f2 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.758479 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2]
I20250112 02:57:38.801482 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf]
I20250112 02:57:38.802983 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952]
I20250112 02:57:38.804272 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041]
I20250112 02:57:38.810516 23305 consensus_queue.cc:1035] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.826226 22143 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:38.828312 22143 ts_tablet_manager.cc:1397] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.111s user 0.012s sys 0.014s
I20250112 02:57:38.830621 23307 raft_consensus.cc:491] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.830641 23246 consensus_queue.cc:1035] T 5d017c05abe94657bfbd2b38329bba16 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.831039 23307 raft_consensus.cc:513] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.833180 22143 raft_consensus.cc:357] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.833626 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62253be16a1a4b2cb7d50264e82478fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:38.834689 21980 leader_election.cc:343] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.834901 23307 leader_election.cc:290] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:38.835801 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62253be16a1a4b2cb7d50264e82478fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:38.843459 21691 consensus_peers.cc:487] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:38.852052 22143 raft_consensus.cc:383] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:38.853247 21978 leader_election.cc:343] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.853587 21978 leader_election.cc:304] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.854911 23308 raft_consensus.cc:2743] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:38.872409 21836 consensus_peers.cc:487] T a2ce633c5b3b4cbea17c171564fa72d6 P cca160b7cffb4ae287ea077e7352ffc9 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2ce633c5b3b4cbea17c171564fa72d6. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:38.872496 22144 tablet_bootstrap.cc:492] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:38.872901 22144 ts_tablet_manager.cc:1397] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.152s user 0.008s sys 0.000s
I20250112 02:57:38.878571 22145 tablet_bootstrap.cc:492] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:38.878964 22145 ts_tablet_manager.cc:1397] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.167s user 0.014s sys 0.039s
I20250112 02:57:38.806365 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab]
I20250112 02:57:38.870661 23246 consensus_queue.cc:1035] T 13edf2c56e8e4254b75e0eb85fb88427 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:38.887779 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac]
I20250112 02:57:38.888388 23246 raft_consensus.cc:491] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.888790 23246 raft_consensus.cc:513] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.892571 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:38.893121 22041 raft_consensus.cc:2463] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:38.895879 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:38.890484 23246 leader_election.cc:290] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.894044 21690 leader_election.cc:304] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
W20250112 02:57:38.903576 21691 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.903879 23246 raft_consensus.cc:2798] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:38.904285 23246 raft_consensus.cc:491] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:38.904633 23246 raft_consensus.cc:3054] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:38.905448 21835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 45514.6us
I20250112 02:57:38.914999 21401 raft_consensus.cc:1235] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 5 FOLLOWER]: Rejecting Update request from peer e76221f7d4ec44da987303d68c393a1b for earlier term 4. Current term is 5. Ops: []
I20250112 02:57:38.890576 22143 raft_consensus.cc:738] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.921156 23246 raft_consensus.cc:513] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.883770 22145 raft_consensus.cc:357] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.928509 22145 raft_consensus.cc:383] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.929113 22145 raft_consensus.cc:738] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.930075 22145 consensus_queue.cc:260] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.922458 22143 consensus_queue.cc:260] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.935303 22143 ts_tablet_manager.cc:1428] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.106s user 0.005s sys 0.000s
I20250112 02:57:38.936798 22144 raft_consensus.cc:357] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.937443 22144 raft_consensus.cc:383] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:38.937680 22144 raft_consensus.cc:738] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:38.939435 22145 ts_tablet_manager.cc:1428] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20250112 02:57:38.941397 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:38.944706 21691 leader_election.cc:343] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:38.945188 23246 leader_election.cc:290] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.942807 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:38.945808 22041 raft_consensus.cc:3054] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.952683 22041 raft_consensus.cc:2463] T be0b4a02597842d3ae1ad0c6197cc808 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:38.959254 21690 leader_election.cc:304] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:38.959695 22143 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:38.967144 22145 tablet_bootstrap.cc:492] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:38.967483 23246 raft_consensus.cc:2798] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:38.916465 23332 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 }, Status: INVALID_TERM, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20250112 02:57:38.977468 23246 raft_consensus.cc:695] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:38.979600 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]
I20250112 02:57:38.978404 23246 consensus_queue.cc:237] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.938251 22144 consensus_queue.cc:260] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.987377 23360 raft_consensus.cc:491] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:38.987771 23360 raft_consensus.cc:513] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:38.989475 23360 leader_election.cc:290] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:38.990670 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:38.991379 22041 raft_consensus.cc:2388] T 685953572d45476ebaf2580fd5e4a3cd P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:38.991349 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685953572d45476ebaf2580fd5e4a3cd" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:38.992939 21835 leader_election.cc:304] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:38.993618 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4]
I20250112 02:57:38.993844 23360 raft_consensus.cc:3054] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:38.996801 22144 ts_tablet_manager.cc:1428] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.124s user 0.007s sys 0.000s
I20250112 02:57:38.998524 22144 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:38.999974 23360 raft_consensus.cc:2743] T 685953572d45476ebaf2580fd5e4a3cd P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.004446 22143 tablet_bootstrap.cc:654] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:38.994553 22145 tablet_bootstrap.cc:654] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.019544 22144 tablet_bootstrap.cc:654] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.017310 23332 consensus_queue.cc:1225] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Peer bcc0757376c3411a9fcbc84ec411befd log is divergent from this leader: its last log entry 3.366 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 365
I20250112 02:57:39.034417 22143 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:39.034839 22143 ts_tablet_manager.cc:1397] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.000s
I20250112 02:57:39.035985 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152]
I20250112 02:57:39.039597 22143 raft_consensus.cc:357] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.040277 22143 raft_consensus.cc:383] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.040521 22143 raft_consensus.cc:738] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.041692 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d]
I20250112 02:57:39.043434 22145 tablet_bootstrap.cc:492] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.038426 23362 raft_consensus.cc:3049] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 LEADER]: Stepping down as leader of term 4
I20250112 02:57:39.043797 22145 ts_tablet_manager.cc:1397] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.007s
I20250112 02:57:39.043787 23362 raft_consensus.cc:738] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Running, Role: LEADER
I20250112 02:57:39.044747 23362 consensus_queue.cc:260] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 4.366, Last appended by leader: 366, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:39.045791 22145 raft_consensus.cc:357] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.046356 22145 raft_consensus.cc:383] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.046598 22145 raft_consensus.cc:738] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.047147 22145 consensus_queue.cc:260] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.049021 22145 ts_tablet_manager.cc:1428] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20250112 02:57:39.049278 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7]
I20250112 02:57:39.047562 23362 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 4 FOLLOWER]: Advancing to term 5
I20250112 02:57:39.051533 22143 consensus_queue.cc:260] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.054093 22144 tablet_bootstrap.cc:492] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:39.054491 22144 ts_tablet_manager.cc:1397] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.002s
I20250112 02:57:39.056666 22144 raft_consensus.cc:357] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.064486 22143 ts_tablet_manager.cc:1428] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250112 02:57:39.064831 22144 raft_consensus.cc:383] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.064500 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]
I20250112 02:57:39.066581 22144 raft_consensus.cc:738] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.075975 22144 consensus_queue.cc:260] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.077718 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_03e04d34d58e463e93d16e858752d3cb [id=30ad19fdfc1246ebbeaf73fff91f925e]
I20250112 02:57:39.082413 22144 ts_tablet_manager.cc:1428] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20250112 02:57:39.086549 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59]
I20250112 02:57:39.091457 22144 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:39.092260 22145 tablet_bootstrap.cc:492] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.100361 22143 tablet_bootstrap.cc:492] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:39.104508 23307 raft_consensus.cc:491] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.105010 23307 raft_consensus.cc:513] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.106494 22145 tablet_bootstrap.cc:654] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.106555 22143 tablet_bootstrap.cc:654] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.107295 23307 leader_election.cc:290] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:39.108909 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034065c6bc435d9f26f85a96297cd3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.110265 21978 leader_election.cc:343] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.113241 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]
I20250112 02:57:39.118544 22143 tablet_bootstrap.cc:492] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:39.118896 22143 ts_tablet_manager.cc:1397] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.005s
I20250112 02:57:39.119261 22144 tablet_bootstrap.cc:654] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.120882 22143 raft_consensus.cc:357] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.121475 22143 raft_consensus.cc:383] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.118670 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c]
I20250112 02:57:39.121690 22143 raft_consensus.cc:738] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.122303 22143 consensus_queue.cc:260] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.125155 22143 ts_tablet_manager.cc:1428] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 02:57:39.125978 22143 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:39.126617 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c]
I20250112 02:57:39.132391 22143 tablet_bootstrap.cc:654] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.132422 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db]
I20250112 02:57:39.134500 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]
I20250112 02:57:39.138103 23246 raft_consensus.cc:491] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.138527 23246 raft_consensus.cc:513] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.140522 23246 leader_election.cc:290] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.142792 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.143523 22041 raft_consensus.cc:2463] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:39.144570 21690 leader_election.cc:304] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:39.145947 23299 raft_consensus.cc:2798] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.146231 23299 raft_consensus.cc:491] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.146492 23299 raft_consensus.cc:3054] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.157848 23299 raft_consensus.cc:513] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.158546 23246 raft_consensus.cc:491] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.161588 23246 raft_consensus.cc:513] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.162091 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5]
I20250112 02:57:39.164274 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]
W20250112 02:57:39.165338 21690 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 247f0ef0ada44d9d957c8ff42b0da4b5
I20250112 02:57:39.165866 23246 leader_election.cc:290] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.169569 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:39.170251 22041 raft_consensus.cc:3054] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.177441 22041 raft_consensus.cc:2463] T f780cf38ac4d41d8b59a3d8882a10030 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.177953 23299 leader_election.cc:290] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.179452 21690 leader_election.cc:304] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:39.180472 23246 raft_consensus.cc:2798] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.180794 22145 tablet_bootstrap.cc:492] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.181170 22145 ts_tablet_manager.cc:1397] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.023s
I20250112 02:57:39.183357 22145 raft_consensus.cc:357] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.183270 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.183946 22145 raft_consensus.cc:383] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.182879 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9034065c6bc435d9f26f85a96297cd3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.184160 22145 raft_consensus.cc:738] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.184710 22145 consensus_queue.cc:260] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:39.185209 21980 leader_election.cc:343] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.185595 21980 leader_election.cc:304] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.186233 22145 ts_tablet_manager.cc:1428] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:57:39.186295 21902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247f0ef0ada44d9d957c8ff42b0da4b5" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.187368 23308 raft_consensus.cc:2743] T b9034065c6bc435d9f26f85a96297cd3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.188261 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:39.189358 21691 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:39.191622 21691 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.191922 21691 leader_election.cc:304] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2; no voters: cca160b7cffb4ae287ea077e7352ffc9, e0406c08e9714e87bdd7fd6b66c9756d
W20250112 02:57:39.192993 21691 leader_election.cc:343] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.193720 23192 raft_consensus.cc:2743] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.195770 23246 raft_consensus.cc:695] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.196619 23246 consensus_queue.cc:237] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.218147 22145 tablet_bootstrap.cc:492] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.219803 23192 raft_consensus.cc:491] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.220212 23192 raft_consensus.cc:513] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.221930 23192 leader_election.cc:290] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.223145 22145 tablet_bootstrap.cc:654] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.224643 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.226430 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.226976 22045 raft_consensus.cc:2463] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
W20250112 02:57:39.227988 21691 leader_election.cc:343] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.229036 21690 leader_election.cc:304] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.222319 23192 consensus_queue.cc:1035] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250112 02:57:39.229846 23298 raft_consensus.cc:2798] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.230710 23298 raft_consensus.cc:491] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.231009 23298 raft_consensus.cc:3054] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.241472 21899 raft_consensus.cc:3054] T ae3ed0bdc2ec44998ee461d4b1254e91 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.242539 23298 raft_consensus.cc:513] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.243182 22144 tablet_bootstrap.cc:492] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:39.243819 22144 ts_tablet_manager.cc:1397] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.153s user 0.010s sys 0.005s
I20250112 02:57:39.245613 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:39.246762 21691 leader_election.cc:343] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.246889 22144 raft_consensus.cc:357] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.247687 22143 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:39.247855 22144 raft_consensus.cc:383] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.248229 22143 ts_tablet_manager.cc:1397] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.122s user 0.008s sys 0.005s
I20250112 02:57:39.251605 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]
I20250112 02:57:39.252974 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:39.253558 22041 raft_consensus.cc:3054] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.254313 22145 tablet_bootstrap.cc:492] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.257344 23298 leader_election.cc:290] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.259760 22145 ts_tablet_manager.cc:1397] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20250112 02:57:39.260715 22143 raft_consensus.cc:357] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.264001 22041 raft_consensus.cc:2463] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.266672 22143 raft_consensus.cc:383] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.269176 22143 raft_consensus.cc:738] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.276960 22143 consensus_queue.cc:260] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.264408 23298 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.278625 23298 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.281761 23192 raft_consensus.cc:491] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.291236 23124 consensus_queue.cc:1035] T 2d793344526a404e8a737e7fb628e521 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:39.298467 23192 raft_consensus.cc:513] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.301033 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:39.302199 21691 leader_election.cc:343] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:39.303018 21378 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.217.188:35683 timed out after 0.545s (SENT) [suppressed 1 similar messages]
I20250112 02:57:39.264604 22145 raft_consensus.cc:357] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.304044 22145 raft_consensus.cc:383] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.304278 22145 raft_consensus.cc:738] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.248112 22144 raft_consensus.cc:738] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.305092 22144 consensus_queue.cc:260] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.304917 22145 consensus_queue.cc:260] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.254254 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d]
I20250112 02:57:39.307355 22145 ts_tablet_manager.cc:1428] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20250112 02:57:39.307821 23192 leader_election.cc:290] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.308073 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f]
I20250112 02:57:39.308210 22145 tablet_bootstrap.cc:492] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.309274 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.309401 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde]
I20250112 02:57:39.312880 21690 leader_election.cc:304] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.313493 23299 raft_consensus.cc:2798] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.313516 22045 raft_consensus.cc:2463] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:39.313925 23299 raft_consensus.cc:695] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.276335 23307 raft_consensus.cc:491] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.316172 23299 consensus_queue.cc:237] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.315930 23307 raft_consensus.cc:513] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.318369 21690 leader_election.cc:304] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.319255 23192 raft_consensus.cc:2798] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.319603 23192 raft_consensus.cc:491] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.319967 22145 tablet_bootstrap.cc:654] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.322872 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.323416 21901 raft_consensus.cc:2463] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.325042 23192 raft_consensus.cc:3054] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.324522 21691 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:39.321825 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698c4b2fd55d43aeb1fd5a3eaf967da5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:39.330961 23124 raft_consensus.cc:2798] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:57:39.331367 23124 raft_consensus.cc:491] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250112 02:57:39.331441 21978 leader_election.cc:343] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.331835 23124 raft_consensus.cc:3054] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Advancing to term 2
W20250112 02:57:39.339749 21690 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 443aa544f6df4d50a78c9effd306e216
I20250112 02:57:39.340302 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698c4b2fd55d43aeb1fd5a3eaf967da5" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:39.341336 21980 leader_election.cc:343] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.341666 21980 leader_election.cc:304] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.342283 23308 raft_consensus.cc:2743] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.343169 23298 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.343592 23307 leader_election.cc:290] T 698c4b2fd55d43aeb1fd5a3eaf967da5 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:39.351326 22143 ts_tablet_manager.cc:1428] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.103s user 0.004s sys 0.001s
I20250112 02:57:39.352033 23124 raft_consensus.cc:513] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.353149 22144 ts_tablet_manager.cc:1428] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.109s user 0.008s sys 0.000s
W20250112 02:57:39.355581 21378 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.188:35683, user_credentials={real_user=slave}} blocked reactor thread for 52602.8us
I20250112 02:57:39.377022 22144 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:39.393909 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a]
I20250112 02:57:39.395445 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3]
I20250112 02:57:39.352639 22143 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:39.400395 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "443aa544f6df4d50a78c9effd306e216" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:39.400965 21901 raft_consensus.cc:3054] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:57:39.365101 23192 raft_consensus.cc:513] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.390811 23246 raft_consensus.cc:491] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.402823 23246 raft_consensus.cc:513] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.403800 23192 leader_election.cc:290] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.404469 23246 leader_election.cc:290] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.404541 23307 raft_consensus.cc:491] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.396768 23298 consensus_queue.cc:1035] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:39.404940 23307 raft_consensus.cc:513] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.406401 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:39.406736 23307 leader_election.cc:290] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.408036 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41369773a0034c65b25fee871e6a25aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.408356 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f5144c9c644a7f8ca95082459c4556" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:39.408788 22045 raft_consensus.cc:3054] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:39.409370 21980 leader_election.cc:343] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.410794 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.411355 22041 raft_consensus.cc:2463] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:39.411783 21901 raft_consensus.cc:2463] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 2.
I20250112 02:57:39.412360 23298 raft_consensus.cc:491] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.412765 23298 raft_consensus.cc:513] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.413904 23124 leader_election.cc:290] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.414587 23298 leader_election.cc:290] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.414878 22143 tablet_bootstrap.cc:654] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.416764 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41369773a0034c65b25fee871e6a25aa" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.416965 21691 leader_election.cc:343] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.417501 21691 leader_election.cc:304] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:39.418617 23299 raft_consensus.cc:2798] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 02:57:39.419025 23299 raft_consensus.cc:695] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [term 2 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.419719 23299 consensus_queue.cc:237] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:39.425890 21978 leader_election.cc:343] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.426271 21978 leader_election.cc:304] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.426203 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.428401 21690 leader_election.cc:304] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:39.428562 23308 raft_consensus.cc:2743] T 41369773a0034c65b25fee871e6a25aa P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.429483 23298 consensus_queue.cc:1035] T 685953572d45476ebaf2580fd5e4a3cd P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:39.430527 23298 raft_consensus.cc:2798] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.430820 23298 raft_consensus.cc:491] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.431093 23298 raft_consensus.cc:3054] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.432965 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.433437 22044 raft_consensus.cc:2463] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:39.436111 22045 raft_consensus.cc:2463] T 68f5144c9c644a7f8ca95082459c4556 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.429809 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:39.448380 21690 leader_election.cc:343] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 443aa544f6df4d50a78c9effd306e216
I20250112 02:57:39.449323 21690 leader_election.cc:304] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:39.449985 21690 leader_election.cc:304] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.450982 23192 raft_consensus.cc:2798] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.451311 23192 raft_consensus.cc:491] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.451642 23192 raft_consensus.cc:3054] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.457919 23192 raft_consensus.cc:513] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.458725 23298 raft_consensus.cc:513] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.461237 23298 leader_election.cc:290] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.459281 23124 raft_consensus.cc:2798] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.463327 23124 raft_consensus.cc:695] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.464349 23124 consensus_queue.cc:237] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.450358 22144 tablet_bootstrap.cc:654] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.461592 23192 leader_election.cc:290] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:39.465657 21691 leader_election.cc:343] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:39.467448 21691 leader_election.cc:343] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.469678 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:39.471585 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:39.475260 21691 leader_election.cc:343] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:39.475808 21691 leader_election.cc:343] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.398447 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd]
I20250112 02:57:39.494289 22145 tablet_bootstrap.cc:492] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.494781 22145 ts_tablet_manager.cc:1397] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.187s user 0.011s sys 0.018s
I20250112 02:57:39.493881 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3]
I20250112 02:57:39.496938 22145 raft_consensus.cc:357] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.497453 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b]
I20250112 02:57:39.497552 22145 raft_consensus.cc:383] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.497784 22145 raft_consensus.cc:738] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.502197 22143 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:39.502665 22143 ts_tablet_manager.cc:1397] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.006s
I20250112 02:57:39.505285 22143 raft_consensus.cc:357] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.506042 22143 raft_consensus.cc:383] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.506147 23362 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.506572 23362 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:39.508258 23362 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:39.509155 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:39.509250 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:39.509757 21401 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 5.
I20250112 02:57:39.510910 21531 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters: e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:39.511605 23362 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250112 02:57:39.511916 23362 raft_consensus.cc:491] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.512215 23362 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 5 FOLLOWER]: Advancing to term 6
I20250112 02:57:39.515858 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:39.516546 22045 raft_consensus.cc:3054] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.518523 23307 raft_consensus.cc:491] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.518914 23307 raft_consensus.cc:513] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.519173 23362 raft_consensus.cc:513] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:39.520588 23307 leader_election.cc:290] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.521875 23362 leader_election.cc:290] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 6 election: Requested vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049)
I20250112 02:57:39.522426 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd"
I20250112 02:57:39.522462 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e76221f7d4ec44da987303d68c393a1b" candidate_term: 6 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217"
I20250112 02:57:39.522975 21401 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 5 FOLLOWER]: Advancing to term 6
I20250112 02:57:39.523965 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486dd5218bad42b0bb9064b7a14f3a4e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.524763 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d]
I20250112 02:57:39.525590 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486dd5218bad42b0bb9064b7a14f3a4e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.525827 21980 leader_election.cc:343] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.526281 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680]
W20250112 02:57:39.526739 21978 leader_election.cc:343] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.527086 21978 leader_election.cc:304] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.528137 23307 raft_consensus.cc:2743] T 486dd5218bad42b0bb9064b7a14f3a4e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.528849 22144 tablet_bootstrap.cc:492] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:39.530115 22144 ts_tablet_manager.cc:1397] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.153s user 0.009s sys 0.002s
I20250112 02:57:39.532460 22144 raft_consensus.cc:357] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.533160 22144 raft_consensus.cc:383] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.533412 22144 raft_consensus.cc:738] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.534075 22144 consensus_queue.cc:260] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.537246 23360 raft_consensus.cc:491] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.537633 23360 raft_consensus.cc:513] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.534849 21401 raft_consensus.cc:2463] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e76221f7d4ec44da987303d68c393a1b in term 6.
I20250112 02:57:39.539563 23360 leader_election.cc:290] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.539773 21531 leader_election.cc:304] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b; no voters: e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:39.540836 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:39.540944 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:39.541410 22038 raft_consensus.cc:3054] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.541538 22044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e922b68d33b4a76a260816475ee6a3c" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.542521 23362 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 6 FOLLOWER]: Leader election won for term 6
I20250112 02:57:39.543292 21836 leader_election.cc:304] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:39.543974 23360 raft_consensus.cc:2743] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.547065 22038 raft_consensus.cc:2463] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.550768 22144 ts_tablet_manager.cc:1428] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250112 02:57:39.551647 22144 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:39.498348 22145 consensus_queue.cc:260] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.559180 22145 ts_tablet_manager.cc:1428] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.064s user 0.007s sys 0.000s
I20250112 02:57:39.536314 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b]
I20250112 02:57:39.537261 22045 raft_consensus.cc:2463] T 334ed386f72d4fca870ade9d6a44589a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:39.566143 21690 leader_election.cc:304] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.568240 23388 raft_consensus.cc:2798] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.568626 23388 raft_consensus.cc:695] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.506297 22143 raft_consensus.cc:738] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.569348 23388 consensus_queue.cc:237] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.573194 23362 raft_consensus.cc:695] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [term 6 LEADER]: Becoming Leader. State: Replica: e76221f7d4ec44da987303d68c393a1b, State: Running, Role: LEADER
I20250112 02:57:39.570052 22143 consensus_queue.cc:260] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.573987 23362 consensus_queue.cc:237] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 4.366, Last appended by leader: 366, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:39.577939 21690 leader_election.cc:304] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.579190 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0]
I20250112 02:57:39.579252 23246 raft_consensus.cc:2798] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.579619 23246 raft_consensus.cc:695] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:39.580268 23246 consensus_queue.cc:237] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.582342 22143 ts_tablet_manager.cc:1428] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.079s user 0.006s sys 0.000s
I20250112 02:57:39.583276 22143 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:39.591022 23308 raft_consensus.cc:491] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.591444 23308 raft_consensus.cc:513] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.593374 23308 leader_election.cc:290] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.593526 22144 tablet_bootstrap.cc:654] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.593942 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db9ff39ff2341199549155a2db0aaba" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.594305 23360 raft_consensus.cc:491] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2770701ca54288956df9d910d7d3d2)
I20250112 02:57:39.594795 23360 raft_consensus.cc:513] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:39.595137 21980 leader_election.cc:343] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.596457 23360 leader_election.cc:290] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.597074 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db9ff39ff2341199549155a2db0aaba" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.598062 21978 leader_election.cc:343] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.598471 21978 leader_election.cc:304] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
W20250112 02:57:39.599380 21835 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:39.599741 23308 raft_consensus.cc:2743] T 1db9ff39ff2341199549155a2db0aaba P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:39.599865 21836 leader_election.cc:343] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:39.600162 21836 leader_election.cc:304] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:39.601331 23360 raft_consensus.cc:2743] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:39.605819 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]
I20250112 02:57:39.608371 23360 raft_consensus.cc:491] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.608803 23360 raft_consensus.cc:513] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:39.613147 21835 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:39.614238 23360 leader_election.cc:290] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:39.614357 21836 leader_election.cc:343] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 4e2b248224f9416faf4bd83a853ec918
I20250112 02:57:39.615593 21836 leader_election.cc:304] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:39.616272 23360 raft_consensus.cc:2743] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.616905 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:39.617609 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:39.618695 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]
I20250112 02:57:39.620852 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7]
I20250112 02:57:39.629638 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_181a7bdabe584055b44baf7ff5ab440f [id=8dbf245ab5354db7a93b54749e978c65]
I20250112 02:57:39.633215 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]
I20250112 02:57:39.637143 22145 tablet_bootstrap.cc:492] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.638656 22144 tablet_bootstrap.cc:492] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:39.639008 22144 ts_tablet_manager.cc:1397] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.000s
I20250112 02:57:39.641158 22144 raft_consensus.cc:357] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.641763 22144 raft_consensus.cc:383] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.641973 22145 tablet_bootstrap.cc:654] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.642019 22144 raft_consensus.cc:738] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.642818 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f]
I20250112 02:57:39.645715 22143 tablet_bootstrap.cc:654] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.656311 23308 raft_consensus.cc:491] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.656729 23308 raft_consensus.cc:513] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.658375 23308 leader_election.cc:290] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.659916 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.660038 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:39.660604 21757 raft_consensus.cc:2463] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
W20250112 02:57:39.661186 21980 leader_election.cc:343] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.661841 21978 leader_election.cc:304] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.662700 23308 raft_consensus.cc:2798] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:39.662966 23308 raft_consensus.cc:491] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:39.663251 23308 raft_consensus.cc:3054] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.666939 22143 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:39.669533 22143 ts_tablet_manager.cc:1397] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.003s
I20250112 02:57:39.670343 23308 raft_consensus.cc:513] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.672463 22144 consensus_queue.cc:260] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.674342 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]
I20250112 02:57:39.675752 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]
I20250112 02:57:39.677706 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2]
I20250112 02:57:39.679971 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
W20250112 02:57:39.681331 21980 leader_election.cc:343] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.683154 23308 leader_election.cc:290] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.683293 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a345af15e7664d01a184193ddb51572b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:39.683859 21757 raft_consensus.cc:3054] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:39.689472 21757 raft_consensus.cc:2463] T a345af15e7664d01a184193ddb51572b P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 1.
I20250112 02:57:39.690378 21978 leader_election.cc:304] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.691437 23308 raft_consensus.cc:2798] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:39.692143 23308 raft_consensus.cc:695] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [term 1 LEADER]: Becoming Leader. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Running, Role: LEADER
I20250112 02:57:39.692847 23308 consensus_queue.cc:237] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.702059 23360 raft_consensus.cc:491] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2770701ca54288956df9d910d7d3d2)
I20250112 02:57:39.702487 23360 raft_consensus.cc:513] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.704190 23360 leader_election.cc:290] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:39.704324 22145 tablet_bootstrap.cc:492] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.704722 22145 ts_tablet_manager.cc:1397] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.000s
W20250112 02:57:39.705816 21836 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:39.712241 23307 raft_consensus.cc:491] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.712607 23307 raft_consensus.cc:513] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.714118 23307 leader_election.cc:290] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.715461 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cacc670998c4452bd3c03d99a62ad82" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:39.716612 21980 leader_election.cc:343] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.717881 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cacc670998c4452bd3c03d99a62ad82" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.719033 21978 leader_election.cc:343] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.719354 21978 leader_election.cc:304] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.720175 23307 raft_consensus.cc:2743] T 1cacc670998c4452bd3c03d99a62ad82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.727166 22144 ts_tablet_manager.cc:1428] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.088s user 0.005s sys 0.001s
W20250112 02:57:39.729049 21835 leader_election.cc:343] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Not found: Tablet not found: 2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:39.731115 21835 leader_election.cc:304] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:39.733016 23360 raft_consensus.cc:2743] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:57:39.739722 21691 consensus_peers.cc:487] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:39.737381 22143 raft_consensus.cc:357] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.749114 22143 raft_consensus.cc:383] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.753971 22145 raft_consensus.cc:357] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.754613 22145 raft_consensus.cc:383] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.754868 22145 raft_consensus.cc:738] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.755425 22145 consensus_queue.cc:260] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.752974 22143 raft_consensus.cc:738] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.756340 22144 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:39.757207 22145 ts_tablet_manager.cc:1428] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.052s user 0.004s sys 0.000s
I20250112 02:57:39.782402 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]
I20250112 02:57:39.768253 22143 consensus_queue.cc:260] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.785032 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2]
I20250112 02:57:39.787176 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158]
I20250112 02:57:39.787464 22144 tablet_bootstrap.cc:654] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.790532 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9]
I20250112 02:57:39.792304 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]
I20250112 02:57:39.795111 22145 tablet_bootstrap.cc:492] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.798770 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268]
I20250112 02:57:39.800205 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]
I20250112 02:57:39.803180 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2]
I20250112 02:57:39.814649 22144 tablet_bootstrap.cc:492] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:39.815121 22144 ts_tablet_manager.cc:1397] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.002s
I20250112 02:57:39.818881 22144 raft_consensus.cc:357] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.819573 22144 raft_consensus.cc:383] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.819833 22144 raft_consensus.cc:738] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.812812 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]
I20250112 02:57:39.823586 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43]
I20250112 02:57:39.820465 22144 consensus_queue.cc:260] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.836544 23308 raft_consensus.cc:491] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.836969 23308 raft_consensus.cc:513] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.839408 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:39.840517 23308 leader_election.cc:290] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:39.841307 22144 ts_tablet_manager.cc:1428] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250112 02:57:39.842180 22144 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:39.842239 22145 tablet_bootstrap.cc:654] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:39.841531 21980 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.843761 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822]
I20250112 02:57:39.844275 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:39.845317 21978 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:39.845672 21978 leader_election.cc:304] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:39.846635 23308 raft_consensus.cc:2743] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.853945 22143 ts_tablet_manager.cc:1428] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.181s user 0.011s sys 0.001s
I20250112 02:57:39.861143 22143 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:39.867334 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]
I20250112 02:57:39.869108 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18]
I20250112 02:57:39.871001 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c]
I20250112 02:57:39.878633 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]
I20250112 02:57:39.879824 22144 tablet_bootstrap.cc:654] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:39.882952 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0]
I20250112 02:57:39.885056 23360 raft_consensus.cc:491] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:39.885524 23360 raft_consensus.cc:513] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.887480 23360 leader_election.cc:290] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:39.889758 21378 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet eab717cb5d794d15b9b51881b86501fc (table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet eab717cb5d794d15b9b51881b86501fc consensus state has no leader
I20250112 02:57:39.891696 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:39.898228 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5f2a4f6a8e4a64a7642d5a18b58a6d" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:39.899507 21835 leader_election.cc:304] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:39.900072 22145 tablet_bootstrap.cc:492] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:39.900429 22145 ts_tablet_manager.cc:1397] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.000s
W20250112 02:57:39.902288 21378 leader_election.cc:336] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683): Timed out: RequestConsensusVote RPC to 127.20.217.188:35683 timed out after 0.545s (SENT)
I20250112 02:57:39.902381 23360 raft_consensus.cc:2743] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:39.902674 21378 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd; no voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217
I20250112 02:57:39.903518 22145 raft_consensus.cc:357] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.904340 22145 raft_consensus.cc:383] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:39.904649 22145 raft_consensus.cc:738] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:39.905277 22145 consensus_queue.cc:260] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:39.923115 23411 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250112 02:57:39.924408 22145 ts_tablet_manager.cc:1428] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20250112 02:57:39.925217 22145 tablet_bootstrap.cc:492] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:39.921566 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 4 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:39.926744 23377 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:39.927111 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:39.927397 23377 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:39.941620 23377 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
W20250112 02:57:39.947165 22865 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:54033 (request call id 54) took 1162 ms (client timeout 1000 ms). Trace:
W20250112 02:57:39.947496 22865 rpcz_store.cc:269] 0112 02:57:38.784368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.784509 (+ 141us) service_pool.cc:225] Handling call
0112 02:57:39.882129 (+1097620us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400040454. stack: 00007fe157858853 00007fe157858750 00007fe1573cd584 00000000004fcd24 00000000004fccb9 00007fe15e758d50 00007fe15e7dbf2a 00007fe160480dec 00007fe160578dee 00007fe15e2d6fc4 00007fe15e2d6f52 00007fe15e2d6e81 00007fe15e2d6dfc 00007fe15e2d60b2 00007fe15c5a0dfc 00007fe15c5a0236
0112 02:57:39.946970 (+ 64841us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2515309440}
I20250112 02:57:39.937182 22881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40982 (request call id 3) took 1208 ms. Trace:
I20250112 02:57:39.953310 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 5 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:39.954030 21401 raft_consensus.cc:2363] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9ef739ef9dc460ca8287b1d2c6c2217 for earlier term 5. Current term is 6.
I20250112 02:57:39.957990 22840 leader_election.cc:400] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 5 pre-election: Vote denied by peer bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9ef739ef9dc460ca8287b1d2c6c2217 for earlier term 5. Current term is 6.
I20250112 02:57:39.958250 22840 leader_election.cc:403] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250112 02:57:39.959053 23377 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 4 FOLLOWER]: Advancing to term 6
I20250112 02:57:39.971343 22143 tablet_bootstrap.cc:654] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:39.977149 22864 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:58767 (request call id 46) took 1220 ms (client timeout 1000 ms). Trace:
W20250112 02:57:39.977568 22864 rpcz_store.cc:269] 0112 02:57:38.732329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.732582 (+ 253us) service_pool.cc:225] Handling call
0112 02:57:39.943805 (+1211223us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400040454. stack: 00007fe157858853 00007fe157858750 00007fe1573cd584 00000000004fcd24 00000000004fccb9 00007fe15e758d50 00007fe15e7dbf2a 00007fe160480dec 00007fe160578dee 00007fe15e2d6fc4 00007fe15e2d6f52 00007fe15e2d6e81 00007fe15e2d6dfc 00007fe15e2d60b2 00007fe15c5a0dfc 00007fe15c5a0236
0112 02:57:39.952581 (+ 8776us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2771835136}
W20250112 02:57:39.981206 22862 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:51299 (request call id 47) took 1142 ms (client timeout 1000 ms). Trace:
I20250112 02:57:39.991394 21901 raft_consensus.cc:3054] T 6e922b68d33b4a76a260816475ee6a3c P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.007445 22145 tablet_bootstrap.cc:654] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:39.985064 22862 rpcz_store.cc:269] 0112 02:57:38.814846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.814986 (+ 140us) service_pool.cc:225] Handling call
0112 02:57:39.946925 (+1131939us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400040454. stack: 00007fe157858853 00007fe157858750 00007fe1573cd584 00000000004fcd24 00000000004fccb9 00007fe15e758d50 00007fe15e7dbf2a 00007fe160480dec 00007fe160578dee 00007fe15e2d6fc4 00007fe15e2d6f52 00007fe15e2d6e81 00007fe15e2d6dfc 00007fe15e2d60b2 00007fe15c5a0dfc 00007fe15c5a0236
0112 02:57:39.956725 (+ 9800us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2576698624}
I20250112 02:57:40.013545 23377 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9ef739ef9dc460ca8287b1d2c6c2217 for earlier term 5. Current term is 6.
I20250112 02:57:40.020622 23308 raft_consensus.cc:491] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.021075 23308 raft_consensus.cc:513] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.022864 23308 leader_election.cc:290] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.024274 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1f4a9244c4b40ad8557de5b36a40e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:40.025588 21978 leader_election.cc:343] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.030442 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ce1f4a9244c4b40ad8557de5b36a40e" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.031533 21980 leader_election.cc:343] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.031847 21980 leader_election.cc:304] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.032660 23308 raft_consensus.cc:2743] T 4ce1f4a9244c4b40ad8557de5b36a40e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.024742 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682]
I20250112 02:57:40.033476 23410 raft_consensus.cc:491] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.033876 23410 raft_consensus.cc:513] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.035825 23410 leader_election.cc:290] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.037863 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247f0ef0ada44d9d957c8ff42b0da4b5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.038384 21757 raft_consensus.cc:2463] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 0.
I20250112 02:57:40.039796 21835 leader_election.cc:304] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
W20250112 02:57:40.040839 21836 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 247f0ef0ada44d9d957c8ff42b0da4b5
I20250112 02:57:40.041539 23410 raft_consensus.cc:2798] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:40.041883 23410 raft_consensus.cc:491] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:40.042232 23410 raft_consensus.cc:3054] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.047813 22145 tablet_bootstrap.cc:492] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:40.048197 22145 ts_tablet_manager.cc:1397] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.123s user 0.008s sys 0.000s
I20250112 02:57:40.050179 22145 raft_consensus.cc:357] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.050729 22145 raft_consensus.cc:383] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.050961 22145 raft_consensus.cc:738] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.054366 23410 raft_consensus.cc:513] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.059371 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "247f0ef0ada44d9d957c8ff42b0da4b5" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2"
I20250112 02:57:40.059846 21757 raft_consensus.cc:3054] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.066147 21757 raft_consensus.cc:2463] T 247f0ef0ada44d9d957c8ff42b0da4b5 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cca160b7cffb4ae287ea077e7352ffc9 in term 1.
I20250112 02:57:40.067220 21835 leader_election.cc:304] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9; no voters:
I20250112 02:57:40.067930 23360 raft_consensus.cc:2798] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:40.075305 22145 consensus_queue.cc:260] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.079835 22878 raft_consensus.cc:1270] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 6 FOLLOWER]: Refusing update from remote peer e76221f7d4ec44da987303d68c393a1b: Log matching property violated. Preceding OpId in replica: term: 4 index: 366. Preceding OpId from leader: term: 6 index: 367. (index mismatch)
I20250112 02:57:40.081693 23395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 365, Time since last communication: 0.000s
I20250112 02:57:40.088078 23377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.089140 23377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.091578 23388 consensus_queue.cc:1035] T be0b4a02597842d3ae1ad0c6197cc808 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.094936 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.095578 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.095963 23351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.096542 23351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.087054 23360 raft_consensus.cc:695] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Running, Role: LEADER
I20250112 02:57:40.098035 23360 consensus_queue.cc:237] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.103094 21401 raft_consensus.cc:1270] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Refusing update from remote peer e76221f7d4ec44da987303d68c393a1b: Log matching property violated. Preceding OpId in replica: term: 3 index: 366. Preceding OpId from leader: term: 6 index: 367. (index mismatch)
I20250112 02:57:40.107306 22144 tablet_bootstrap.cc:492] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:40.107465 22145 ts_tablet_manager.cc:1428] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20250112 02:57:40.107709 22144 ts_tablet_manager.cc:1397] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.266s user 0.005s sys 0.007s
W20250112 02:57:40.110328 21836 leader_election.cc:343] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 247f0ef0ada44d9d957c8ff42b0da4b5
I20250112 02:57:40.112486 22143 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:40.112847 22143 ts_tablet_manager.cc:1397] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.254s user 0.008s sys 0.006s
I20250112 02:57:40.110713 22144 raft_consensus.cc:357] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.114578 23410 leader_election.cc:290] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.119138 22143 raft_consensus.cc:357] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.119709 22143 raft_consensus.cc:383] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.119930 22143 raft_consensus.cc:738] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.126557 22145 tablet_bootstrap.cc:492] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:40.126677 21901 raft_consensus.cc:3054] T be5f2a4f6a8e4a64a7642d5a18b58a6d P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.130072 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]
I20250112 02:57:40.131551 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f]
I20250112 02:57:40.132339 23395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 365, Time since last communication: 0.000s
I20250112 02:57:40.132838 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]
I20250112 02:57:40.132973 23395 consensus_queue.cc:1225] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [LEADER]: Peer bcc0757376c3411a9fcbc84ec411befd log is divergent from this leader: its last log entry 3.366 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 365
I20250112 02:57:40.134033 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143]
I20250112 02:57:40.135129 21401 pending_rounds.cc:77] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Aborting all ops after (but not including) 365
I20250112 02:57:40.135442 21401 pending_rounds.cc:99] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Aborting uncommitted WRITE_OP operation due to leader change: 3.366
I20250112 02:57:40.135073 22143 consensus_queue.cc:260] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:40.154656 21533 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.189:36049, user_credentials={real_user=slave}} blocked reactor thread for 54467.4us
I20250112 02:57:40.114096 22144 raft_consensus.cc:383] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.158756 22144 raft_consensus.cc:738] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.160475 23308 raft_consensus.cc:491] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.159839 22144 consensus_queue.cc:260] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.160961 23308 raft_consensus.cc:513] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.163503 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1144ac759a4db496f67cdc9ec35020" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.163831 21378 leader_election.cc:336] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683): Timed out: RequestConsensusVote RPC to 127.20.217.188:35683 timed out after 0.716s (SENT)
W20250112 02:57:40.164891 21980 leader_election.cc:343] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.166077 23308 leader_election.cc:290] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.178254 23386 consensus_queue.cc:1035] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.167908 23395 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.182085 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86]
I20250112 02:57:40.182480 23395 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
I20250112 02:57:40.187052 21899 raft_consensus.cc:3054] T f780cf38ac4d41d8b59a3d8882a10030 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.187283 23411 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e76221f7d4ec44da987303d68c393a1b. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.188161 23411 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.192520 23431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.195322 23431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.200057 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4]
I20250112 02:57:40.201393 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815]
I20250112 02:57:40.201799 23386 raft_consensus.cc:491] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.202204 23386 raft_consensus.cc:513] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.203054 22144 ts_tablet_manager.cc:1428] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.095s user 0.006s sys 0.004s
I20250112 02:57:40.203814 23386 leader_election.cc:290] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.209378 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a1144ac759a4db496f67cdc9ec35020" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.211127 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:40.211723 22038 raft_consensus.cc:2463] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:40.215263 21690 leader_election.cc:304] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
E20250112 02:57:40.138236 21391 catalog_manager.cc:5681] Error updating tablets from cca160b7cffb4ae287ea077e7352ffc9: Aborted: Op aborted by new leader. Tablet report was: is_incremental: true updated_tablets { tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a2ce633c5b3b4cbea17c171564fa72d6" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "443aa544f6df4d50a78c9effd306e216" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0df734ba87264abd803237a5bbb32da2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4e2b248224f9416faf4bd83a853ec918" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f445a4bf43db4571b6d3a4afe8fe823a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "46f8052651fa496cbb3702c17f592106" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "82825c2061d74cafa814912bec005654" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1f21e3c769364b58bd371aac338db1f2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } updated_tablets { tablet_id: "7af7e02e5fc248839b434f42f95c6509" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } updated_tablets { tablet_id: "368af02998984cb2bee487b15b6a2b35" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e8f7301bed6c4d98a80c12b70073e62f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e21a0e4b5ebc426383cf332f46492b80" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2601daad8ba742ec88ccfe603440e8e7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e0977a10252840a381234f02bb7339b3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "67260685978247898ad224128ed8964f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "557de3de9a4d4d0bac1dc16f66ab6b01" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } updated_tablets { tablet_id: "9a8273b629804f1c86a813b04747a1aa" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "eef30508946e4376961b6cd463172c1f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "82d2ac91c5554a64936f364e4e1978f8" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27069 live_row_count: 0 } } updated_tablets { tablet_id: "1800d0eb7a9a48d295aca16607291add" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "cf8bf172abad43ac9a519310334d1640" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "65eb8c99917247edbdbf5de42f3b505c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f08c1affbaf247059ab65222e2e95bab" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f47e14720a704a19ac3aea8723e13c28" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "118ef6d7c16941598177b92a94c4b472" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "22aac4332d0e476d9758004972ef8ab2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c80b7813b50d4739a76776d5d16434f3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "21dc0755b38746a39bd773780e48cda5" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "aab2cf8bc28b47239eaa3dcaa4c90e9c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f663e183c2ed49b98327b2ad0de12ed9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "b4dddfea8e8b49189186930fce46ecc5" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } removed_tablet_ids: "84f7700b8cce4261a32f2009d1481b99" removed_tablet_ids: "9543a0a0359e4b72bd30f1cb6290f767" removed_tablet_ids: "15c16031f5b74cf9885fac110f445e1a" removed_tablet_ids: "c48d1bc147ba48ebbebfd707dc44fe02" removed_tablet_ids: "96f98f6840e24f3c9d03937c1bbc00e7" removed_tablet_ids: "994e1d114e5a4de5aa1d7651907d718e" removed_tablet_ids: "98aa732d27f144aaad088ecc072837e9" removed_tablet_ids: "36495fb158a440309eddedd3910a7ad3" removed_tablet_ids: "27f0d6a5afd140a8af77a5a39dc39dd2" removed_tablet_ids: "6fbe0e20a157479e9160fc6782e3bf37" removed_tablet_ids: "5e3890251e644c1eb596de043b63491e" removed_tablet_ids: "a4b543ed18b34cd58a1c588ed7d079c5" removed_tablet_ids: "3c4998c034e54b1bae0ef132cef5b04d" removed_tablet_ids: "99e81199f79d46d6922eb1c6805d25da" sequence_number: 44
I20250112 02:57:40.217931 23246 raft_consensus.cc:2798] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:40.218255 23246 raft_consensus.cc:491] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:40.218586 23246 raft_consensus.cc:3054] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.218931 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0]
I20250112 02:57:40.219261 21896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:40.219992 23432 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
W20250112 02:57:40.220760 21978 leader_election.cc:343] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.221107 21978 leader_election.cc:304] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.220875 23432 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:57:40.221468 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718]
I20250112 02:57:40.221927 23308 raft_consensus.cc:2743] T 7a1144ac759a4db496f67cdc9ec35020 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.229343 23246 raft_consensus.cc:513] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.232734 22144 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:40.233134 23307 raft_consensus.cc:491] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.233547 23307 raft_consensus.cc:513] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.235383 23307 leader_election.cc:290] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.236004 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb8cbaa79c84aa99bab0a0bd03bbee8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:40.237668 22144 tablet_bootstrap.cc:654] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:40.237619 21980 leader_election.cc:343] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.241231 23246 leader_election.cc:290] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.250236 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:40.250840 22045 raft_consensus.cc:3054] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:40.268235 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 46) took 2745 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.268558 21391 rpcz_store.cc:269] 0112 02:57:37.522864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:37.522966 (+ 102us) service_pool.cc:225] Handling call
0112 02:57:40.268168 (+2745202us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0112 02:57:38.171151 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.171558 (+ 407us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.171585 (+ 27us) write_op.cc:435] Acquired schema lock
0112 02:57:38.171601 (+ 16us) tablet.cc:659] Decoding operations
0112 02:57:38.171947 (+ 346us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.171997 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.172022 (+ 25us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:38.172166 (+ 144us) tablet.cc:698] Row locks acquired
0112 02:57:38.172181 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:38.172301 (+ 120us) write_op.cc:270] Start()
0112 02:57:38.172386 (+ 85us) write_op.cc:276] Timestamp: P: 1736650658172363 usec, L: 0
0112 02:57:38.172404 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:38.172944 (+ 540us) log.cc:844] Serialized 536 byte log entry
0112 02:57:40.135871 (+1962927us) op_driver.cc:464] REPLICATION: finished
0112 02:57:40.135957 (+ 86us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 02:57:40.136002 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.136171 (+ 169us) write_op.cc:454] Released schema lock
0112 02:57:40.136211 (+ 40us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":39,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":18707,"prepare.run_cpu_time_us":2076,"prepare.run_wall_time_us":69462,"raft.queue_time_us":64526,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":629,"replication_time_us":1963301,"thread_start_us":83479,"threads_started":1}]]}
I20250112 02:57:40.270337 23439 tablet.cc:1721] T 9a8273b629804f1c86a813b04747a1aa P 1c2770701ca54288956df9d910d7d3d2: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250112 02:57:40.273466 21691 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.276983 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:40.241329 22143 ts_tablet_manager.cc:1428] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.128s user 0.006s sys 0.001s
I20250112 02:57:40.279014 22143 tablet_bootstrap.cc:492] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
W20250112 02:57:40.279112 21690 consensus_peers.cc:487] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 443aa544f6df4d50a78c9effd306e216. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:40.244897 23438 consensus_queue.cc:1035] T f780cf38ac4d41d8b59a3d8882a10030 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:57:40.306766 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]
I20250112 02:57:40.318094 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb8cbaa79c84aa99bab0a0bd03bbee8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.336930 22145 tablet_bootstrap.cc:654] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:40.348659 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779]
I20250112 02:57:40.352021 22143 tablet_bootstrap.cc:654] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:40.361908 21978 leader_election.cc:343] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.362409 21978 leader_election.cc:304] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.363667 23307 raft_consensus.cc:2743] T 3cb8cbaa79c84aa99bab0a0bd03bbee8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.364288 23307 raft_consensus.cc:491] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.364717 23307 raft_consensus.cc:513] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.365278 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481]
I20250112 02:57:40.368012 23307 leader_election.cc:290] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:40.371038 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01590521b6e4ddc9d7fcc7142d49930" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.372396 21980 leader_election.cc:343] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.373699 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]
I20250112 02:57:40.376758 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e]
I20250112 02:57:40.386371 23410 raft_consensus.cc:491] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.386893 23410 raft_consensus.cc:513] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.389118 23410 leader_election.cc:290] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.390020 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:40.390703 22038 raft_consensus.cc:2388] T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cca160b7cffb4ae287ea077e7352ffc9 in current term 1: Already voted for candidate 1c2770701ca54288956df9d910d7d3d2 in this term.
I20250112 02:57:40.392621 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.402180 23451 tablet.cc:1721] T 9a8273b629804f1c86a813b04747a1aa P e0406c08e9714e87bdd7fd6b66c9756d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:40.407588 23307 raft_consensus.cc:491] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.414930 23308 raft_consensus.cc:491] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.415500 23308 raft_consensus.cc:513] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.417237 23308 leader_election.cc:290] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.417532 22145 tablet_bootstrap.cc:492] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:40.417897 22145 ts_tablet_manager.cc:1397] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.292s user 0.013s sys 0.000s
I20250112 02:57:40.418803 23307 raft_consensus.cc:513] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.420166 22145 raft_consensus.cc:357] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.420755 22145 raft_consensus.cc:383] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.420977 22145 raft_consensus.cc:738] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.421427 23307 leader_election.cc:290] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.422052 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8d7d76e5184d668e4add9810fab7df" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:40.424326 21835 leader_election.cc:304] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:40.426628 23410 raft_consensus.cc:3054] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.430945 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfa495bfefc46b998c2f870d9014900" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.432158 21691 leader_election.cc:343] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.436589 23410 raft_consensus.cc:2743] T 030d48c4421e45f0900705c4b6a4f4ed P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.440438 21901 raft_consensus.cc:1270] T 443aa544f6df4d50a78c9effd306e216 P cca160b7cffb4ae287ea077e7352ffc9 [term 2 FOLLOWER]: Refusing update from remote peer 1c2770701ca54288956df9d910d7d3d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250112 02:57:40.453033 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169]
I20250112 02:57:40.454293 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac]
I20250112 02:57:40.456908 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb]
I20250112 02:57:40.458045 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]
W20250112 02:57:40.463984 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 47) took 2733 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.464545 21391 rpcz_store.cc:269] 0112 02:57:37.730596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.463544 (+2732948us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:40.463763 (+ 219us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250112 02:57:40.465584 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 42) took 2048 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.465888 21391 rpcz_store.cc:269] 0112 02:57:38.417572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.465420 (+2047848us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:40.465531 (+ 111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250112 02:57:40.344810 22045 raft_consensus.cc:2463] T 7c9b3e6c84484575acd46e61dd1ea28a P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:40.468194 23452 tablet.cc:1721] T 9a8273b629804f1c86a813b04747a1aa P cca160b7cffb4ae287ea077e7352ffc9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250112 02:57:40.421496 22145 consensus_queue.cc:260] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:40.423559 21980 leader_election.cc:343] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:40.471089 21980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.130:36673, user_credentials={real_user=slave}} blocked reactor thread for 47580.1us
W20250112 02:57:40.471704 21980 leader_election.cc:343] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.474874 23388 raft_consensus.cc:491] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.475294 23388 raft_consensus.cc:513] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.476879 23388 leader_election.cc:290] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.478690 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c01590521b6e4ddc9d7fcc7142d49930" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:40.471650 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 47) took 1871 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.479669 21391 rpcz_store.cc:269] 0112 02:57:38.600727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.471507 (+1870780us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:40.471609 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250112 02:57:40.479938 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edfa495bfefc46b998c2f870d9014900" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:40.480345 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 48) took 1568 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.480612 21391 rpcz_store.cc:269] 0112 02:57:38.911774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.480211 (+1568437us) service_pool.cc:212] Skipping call since client already timed out
0112 02:57:40.480302 (+ 91us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250112 02:57:40.481087 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a8d7d76e5184d668e4add9810fab7df" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.481707 22143 tablet_bootstrap.cc:492] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:40.482157 22143 ts_tablet_manager.cc:1397] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.203s user 0.005s sys 0.007s
I20250112 02:57:40.484279 22143 raft_consensus.cc:357] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.487157 23442 consensus_queue.cc:1035] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.484892 22143 raft_consensus.cc:383] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.488444 22143 raft_consensus.cc:738] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.488915 22143 consensus_queue.cc:260] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.497390 22144 tablet_bootstrap.cc:492] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:40.497735 22144 ts_tablet_manager.cc:1397] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.265s user 0.006s sys 0.013s
I20250112 02:57:40.499840 22144 raft_consensus.cc:357] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.506913 22144 raft_consensus.cc:383] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.507184 22144 raft_consensus.cc:738] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.507730 22144 consensus_queue.cc:260] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:40.511023 21978 leader_election.cc:343] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.511433 21978 leader_election.cc:304] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.512347 23307 raft_consensus.cc:2743] T c01590521b6e4ddc9d7fcc7142d49930 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.510432 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:40.515214 22144 ts_tablet_manager.cc:1428] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
W20250112 02:57:40.515424 21978 leader_election.cc:343] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.515714 21978 leader_election.cc:304] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.515990 22144 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:40.516674 23307 raft_consensus.cc:2743] T edfa495bfefc46b998c2f870d9014900 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:40.517534 21978 leader_election.cc:343] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.517866 21978 leader_election.cc:304] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.518716 23307 raft_consensus.cc:2743] T 8a8d7d76e5184d668e4add9810fab7df P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.486348 21690 leader_election.cc:304] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.532960 22145 ts_tablet_manager.cc:1428] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.115s user 0.009s sys 0.001s
I20250112 02:57:40.533735 22145 tablet_bootstrap.cc:492] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:40.535137 21690 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 48883.2us
I20250112 02:57:40.536586 23444 raft_consensus.cc:2798] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:40.536957 23444 raft_consensus.cc:695] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:40.537609 23444 consensus_queue.cc:237] T 7c9b3e6c84484575acd46e61dd1ea28a P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.538439 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:40.538635 23446 consensus_queue.cc:1035] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.541656 22045 raft_consensus.cc:2463] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 0.
I20250112 02:57:40.543139 23448 consensus_queue.cc:1035] T 443aa544f6df4d50a78c9effd306e216 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.543373 23446 consensus_queue.cc:1035] T 030d48c4421e45f0900705c4b6a4f4ed P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.548049 21690 leader_election.cc:304] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:40.549067 23448 raft_consensus.cc:2798] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:40.549357 23448 raft_consensus.cc:491] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:40.549624 23448 raft_consensus.cc:3054] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Advancing to term 1
W20250112 02:57:40.550989 21691 consensus_peers.cc:487] T 68f5144c9c644a7f8ca95082459c4556 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:40.553941 21980 consensus_peers.cc:487] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:40.554869 21691 leader_election.cc:343] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.550055 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]). Expected version 1 got 0
I20250112 02:57:40.568243 22143 ts_tablet_manager.cc:1428] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2: Time spent starting tablet: real 0.086s user 0.002s sys 0.002s
I20250112 02:57:40.569051 22143 tablet_bootstrap.cc:492] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2: Bootstrap starting.
I20250112 02:57:40.604674 23448 raft_consensus.cc:513] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.605450 23423 sys_catalog.cc:455] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "e76221f7d4ec44da987303d68c393a1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } } }
I20250112 02:57:40.606321 23423 sys_catalog.cc:458] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b [sys.catalog]: This master's current role is: LEADER
W20250112 02:57:40.556238 22841 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.20.217.188:35683 timed out after 0.612s (SENT)
I20250112 02:57:40.614157 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c]
W20250112 02:57:40.621549 21691 consensus_peers.cc:487] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:40.626772 21386 catalog_manager.cc:5526] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d reported cstate change: term changed from 0 to 1, leader changed from <none> to e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131). New cstate: current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } }
W20250112 02:57:40.635417 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 43) took 990 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.630194 22841 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.217.188:35683, user_credentials={real_user=slave}} blocked reactor thread for 74228.4us
W20250112 02:57:40.635725 21391 rpcz_store.cc:269] 0112 02:57:39.645450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.481134 (+835684us) service_pool.cc:225] Handling call
0112 02:57:40.635357 (+154223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:40.639603 22145 tablet_bootstrap.cc:654] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:40.643373 23457 log.cc:926] Time spent T 030d48c4421e45f0900705c4b6a4f4ed P e0406c08e9714e87bdd7fd6b66c9756d: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
W20250112 02:57:40.646508 21691 consensus_peers.cc:487] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:40.657581 22143 tablet_bootstrap.cc:654] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:40.658604 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48]
I20250112 02:57:40.659880 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330]
I20250112 02:57:40.665310 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:40.667631 21900 raft_consensus.cc:3054] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.674654 22045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d8db14a8054ae69daf42dd76f35511" candidate_uuid: "1c2770701ca54288956df9d910d7d3d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d"
I20250112 02:57:40.675163 22045 raft_consensus.cc:3054] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.680680 22045 raft_consensus.cc:2463] T e7d8db14a8054ae69daf42dd76f35511 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2770701ca54288956df9d910d7d3d2 in term 1.
I20250112 02:57:40.668059 23448 leader_election.cc:290] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.683457 21690 leader_election.cc:304] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters:
I20250112 02:57:40.714493 23307 raft_consensus.cc:491] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.715163 23428 raft_consensus.cc:2798] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:57:40.715029 23307 raft_consensus.cc:513] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.715644 23428 raft_consensus.cc:695] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2770701ca54288956df9d910d7d3d2, State: Running, Role: LEADER
I20250112 02:57:40.716781 23307 leader_election.cc:290] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.716409 23428 consensus_queue.cc:237] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.718720 21901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9f8cbfcd4c41e2bdf8c04216a318bf" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.720070 21980 leader_election.cc:343] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.722558 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_00c60840930f46c49bf80fa7e4489419 [id=8dbf245ab5354db7a93b54749e978c65]). Expected version 1 got 0
I20250112 02:57:40.723543 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]). Expected version 1 got 0
I20250112 02:57:40.724321 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]). Expected version 1 got 0
I20250112 02:57:40.724874 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]). Expected version 1 got 0
I20250112 02:57:40.725759 23448 consensus_queue.cc:1035] T 334ed386f72d4fca870ade9d6a44589a P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.745457 22144 tablet_bootstrap.cc:654] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:40.750926 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet e7d8db14a8054ae69daf42dd76f35511 (table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718]). Expected version 1 got 0
I20250112 02:57:40.751538 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 13edf2c56e8e4254b75e0eb85fb88427 (table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]). Expected version 1 got 0
I20250112 02:57:40.767758 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]
W20250112 02:57:40.766294 21691 leader_election.cc:343] T e7d8db14a8054ae69daf42dd76f35511 P 1c2770701ca54288956df9d910d7d3d2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.772490 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 685953572d45476ebaf2580fd5e4a3cd (table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]). Expected version 1 got 0
I20250112 02:57:40.773416 21386 catalog_manager.cc:5618] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307) does not have the latest schema for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740]). Expected version 1 got 0
I20250112 02:57:40.778362 22145 tablet_bootstrap.cc:492] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
W20250112 02:57:40.791015 21384 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a69123c4ab384e0380c62e52b9f3b42d consensus state has no leader
I20250112 02:57:40.791697 21384 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:50642 (request call id 300) took 4995 ms. Trace:
I20250112 02:57:40.792021 21384 rpcz_store.cc:276] 0112 02:57:35.796718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.796853 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:35.798007 (+ 1154us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:35.798854 (+ 847us) catalog_manager.cc:3627] Apply alter schema
0112 02:57:35.799515 (+ 661us) catalog_manager.cc:3659] Acquired catalog manager lock
0112 02:57:35.799740 (+ 225us) catalog_manager.cc:3817] Updating metadata on disk
0112 02:57:35.888107 (+ 88367us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.888133 (+ 26us) write_op.cc:454] Released schema lock
0112 02:57:35.888361 (+ 228us) catalog_manager.cc:3848] Committing alterations to in-memory state
0112 02:57:40.791568 (+4903207us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.801084 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.801496 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.801525 (+ 29us) write_op.cc:435] Acquired schema lock
0112 02:57:35.801537 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:35.801808 (+ 271us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.801846 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.801869 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:35.802010 (+ 141us) tablet.cc:698] Row locks acquired
0112 02:57:35.802026 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:57:35.802136 (+ 110us) write_op.cc:270] Start()
0112 02:57:35.802220 (+ 84us) write_op.cc:276] Timestamp: P: 1736650655802130 usec, L: 0
0112 02:57:35.802239 (+ 19us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.802767 (+ 528us) log.cc:844] Serialized 405 byte log entry
0112 02:57:35.842734 (+ 39967us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.843025 (+ 291us) write_op.cc:301] APPLY: starting
0112 02:57:35.843068 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.843145 (+ 77us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.843224 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.843372 (+ 148us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.843485 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:35.843532 (+ 47us) write_op.cc:312] APPLY: finished
0112 02:57:35.843685 (+ 153us) log.cc:844] Serialized 23 byte log entry
0112 02:57:35.843792 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.843841 (+ 49us) write_op.cc:454] Released schema lock
0112 02:57:35.843869 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158,"apply.run_cpu_time_us":977,"apply.run_wall_time_us":977,"num_ops":1,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":1996,"prepare.run_wall_time_us":1994,"raft.queue_time_us":269,"raft.run_cpu_time_us":789,"raft.run_wall_time_us":788,"replication_time_us":40417}]]}
I20250112 02:57:40.798028 21350 master-stress-test.cc:579] Killed test thread 5/10
I20250112 02:57:40.798354 21350 master-stress-test.cc:577] Killing test thread 6/10
I20250112 02:57:40.798588 21350 master-stress-test.cc:579] Killed test thread 6/10
I20250112 02:57:40.798768 21350 master-stress-test.cc:577] Killing test thread 7/10
I20250112 02:57:40.790021 22145 ts_tablet_manager.cc:1397] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.256s user 0.011s sys 0.003s
I20250112 02:57:40.807044 21389 catalog_manager.cc:6071] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682] alter complete (version 1)
I20250112 02:57:40.817723 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]
I20250112 02:57:40.825719 23308 raft_consensus.cc:491] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.826220 23308 raft_consensus.cc:513] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.828137 23308 leader_election.cc:290] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673)
I20250112 02:57:40.828526 23308 raft_consensus.cc:491] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:40.829847 23308 raft_consensus.cc:513] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.834589 23360 raft_consensus.cc:491] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1c2770701ca54288956df9d910d7d3d2)
I20250112 02:57:40.835063 23360 raft_consensus.cc:513] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.836076 23308 leader_election.cc:290] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:40.837566 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:40.837613 23442 consensus_queue.cc:1035] T 922b22f0b1284bc4a732371a61cb1ed4 P 1c2770701ca54288956df9d910d7d3d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.838951 22038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:40.839741 23360 leader_election.cc:290] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:40.841177 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3da552a973945e284a386304f182777" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
W20250112 02:57:40.842335 21980 leader_election.cc:343] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.843429 21835 leader_election.cc:304] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:40.844798 23410 raft_consensus.cc:2743] T f3da552a973945e284a386304f182777 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:57:40.852715 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9f8cbfcd4c41e2bdf8c04216a318bf" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.853029 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aff0623f504bf09011ca6de28a3b82" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.854265 21750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2770701ca54288956df9d910d7d3d2" is_pre_election: true
I20250112 02:57:40.854804 21750 raft_consensus.cc:2463] T 750246f0a0c549229d4e7a87cd6ddfd9 P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0406c08e9714e87bdd7fd6b66c9756d in term 0.
W20250112 02:57:40.856669 21978 leader_election.cc:343] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.857007 21978 leader_election.cc:304] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.861061 23307 raft_consensus.cc:2743] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.870602 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f]
W20250112 02:57:40.888123 21978 leader_election.cc:343] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.894877 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4aff0623f504bf09011ca6de28a3b82" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:40.896150 21980 leader_election.cc:343] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:40.896502 21980 leader_election.cc:304] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.871838 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12]
I20250112 02:57:40.900727 21978 leader_election.cc:304] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d; no voters: cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:40.901523 23307 consensus_queue.cc:1035] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:57:40.902758 22143 tablet_bootstrap.cc:492] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2: No bootstrap required, opened a new log
I20250112 02:57:40.902699 23307 raft_consensus.cc:2798] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:57:40.903121 22143 ts_tablet_manager.cc:1397] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2: Time spent bootstrapping tablet: real 0.334s user 0.012s sys 0.005s
I20250112 02:57:40.903079 23307 raft_consensus.cc:491] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:57:40.903440 23307 raft_consensus.cc:3054] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:57:40.905361 22143 raft_consensus.cc:357] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.906057 22143 raft_consensus.cc:383] T 5e9f8cbfcd4c41e2bdf8c04216a318bf P 1c2770701ca54288956df9d910d7d3d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:40.914366 21379 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_181a7bdabe584055b44baf7ff5ab440f [id=8dbf245ab5354db7a93b54749e978c65], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet a69123c4ab384e0380c62e52b9f3b42d consensus state has no leader
I20250112 02:57:40.919868 21385 catalog_manager.cc:5813] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: Sending DeleteTablet for 3 replicas of tablet 46f8052651fa496cbb3702c17f592106
I20250112 02:57:40.921389 21385 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50642 (request call id 290) took 8177 ms. Trace:
I20250112 02:57:40.922313 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f8052651fa496cbb3702c17f592106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:40.921646 21385 rpcz_store.cc:276] 0112 02:57:32.744256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.289715 (+2545459us) service_pool.cc:225] Handling call
0112 02:57:35.290875 (+ 1160us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0112 02:57:35.319243 (+ 28368us) catalog_manager.cc:2753] Modifying in-memory table state
0112 02:57:35.319357 (+ 114us) catalog_manager.cc:2762] Locking tablets
0112 02:57:35.319870 (+ 513us) catalog_manager.cc:2776] Removing table and tablets from system table
0112 02:57:35.393990 (+ 74120us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.394018 (+ 28us) write_op.cc:454] Released schema lock
0112 02:57:35.394242 (+ 224us) catalog_manager.cc:2794] Removing table from by-name map
0112 02:57:35.491337 (+ 97095us) catalog_manager.cc:2810] Committing in-memory state
0112 02:57:40.919637 (+5428300us) catalog_manager.cc:2814] Aborting table tasks
0112 02:57:40.921322 (+ 1685us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:35.375536 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:35.375957 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:35.375982 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:35.375994 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:35.376356 (+ 362us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:35.376396 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:35.376421 (+ 25us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:35.376702 (+ 281us) tablet.cc:698] Row locks acquired
0112 02:57:35.376714 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:35.376861 (+ 147us) write_op.cc:270] Start()
0112 02:57:35.376950 (+ 89us) write_op.cc:276] Timestamp: P: 1736650655376887 usec, L: 0
0112 02:57:35.376967 (+ 17us) op_driver.cc:348] REPLICATION: starting
0112 02:57:35.377470 (+ 503us) log.cc:844] Serialized 774 byte log entry
0112 02:57:35.385655 (+ 8185us) op_driver.cc:464] REPLICATION: finished
0112 02:57:35.386653 (+ 998us) write_op.cc:301] APPLY: starting
0112 02:57:35.386704 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:35.386814 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:35.386914 (+ 100us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:35.387151 (+ 237us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:35.387296 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:35.387330 (+ 34us) write_op.cc:312] APPLY: finished
0112 02:57:35.387468 (+ 138us) log.cc:844] Serialized 29 byte log entry
0112 02:57:35.387580 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:35.387638 (+ 58us) write_op.cc:454] Released schema lock
0112 02:57:35.387664 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":857,"apply.run_cpu_time_us":1190,"apply.run_wall_time_us":1188,"num_ops":2,"prepare.queue_time_us":53866,"prepare.run_cpu_time_us":2190,"prepare.run_wall_time_us":2190,"raft.queue_time_us":215,"raft.run_cpu_time_us":542,"raft.run_wall_time_us":542,"replication_time_us":8596}]]}
I20250112 02:57:40.931092 21350 master-stress-test.cc:579] Killed test thread 7/10
I20250112 02:57:40.931344 21350 master-stress-test.cc:577] Killing test thread 8/10
I20250112 02:57:40.931628 21350 master-stress-test.cc:579] Killed test thread 8/10
I20250112 02:57:40.931814 21350 master-stress-test.cc:577] Killing test thread 9/10
I20250112 02:57:40.932081 21350 master-stress-test.cc:579] Killed test thread 9/10
I20250112 02:57:40.932252 21350 master-stress-test.cc:577] Killing test thread 10/10
I20250112 02:57:40.932516 21350 master-stress-test.cc:579] Killed test thread 10/10
I20250112 02:57:40.932824 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21628
I20250112 02:57:40.911449 23472 raft_consensus.cc:2743] T b4aff0623f504bf09011ca6de28a3b82 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:40.939955 22145 raft_consensus.cc:357] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:40.945679 21388 catalog_manager.cc:6064] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: An error occurred while updating sys-tables: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER
W20250112 02:57:40.950860 21389 catalog_manager.cc:6064] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: An error occurred while updating sys-tables: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER
I20250112 02:57:40.950383 22145 raft_consensus.cc:383] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:40.952390 22145 raft_consensus.cc:738] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:40.955132 22145 consensus_queue.cc:260] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:40.962050 21391 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 48) took 1227 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.968799 21390 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 45) took 6255 ms (client timeout 1000 ms). Trace:
W20250112 02:57:40.962428 21391 rpcz_store.cc:269] 0112 02:57:39.734423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.698687 (+964264us) service_pool.cc:225] Handling call
0112 02:57:40.961968 (+263281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:40.974431 23307 raft_consensus.cc:513] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:40.980258 23475 tablet_replica.cc:331] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:40.984040 22144 tablet_bootstrap.cc:492] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:40.984453 22144 ts_tablet_manager.cc:1397] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.469s user 0.002s sys 0.016s
I20250112 02:57:40.983526 23475 raft_consensus.cc:2238] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:40.995216 22144 raft_consensus.cc:357] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.003666 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f8052651fa496cbb3702c17f592106 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:40.991539 23475 raft_consensus.cc:2267] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.010346 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 994e1d114e5a4de5aa1d7651907d718e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:25 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.012738 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e3890251e644c1eb596de043b63491e with delete_type TABLET_DATA_DELETED (replaced by tablet 78f03a1dfb604b33a0528c28ade287f2) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:40.995776 22145 ts_tablet_manager.cc:1428] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.192s user 0.010s sys 0.002s
I20250112 02:57:41.015496 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 96f98f6840e24f3c9d03937c1bbc00e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d793344526a404e8a737e7fb628e521) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.017256 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 84f7700b8cce4261a32f2009d1481b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:23 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.018010 22144 raft_consensus.cc:383] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.018255 22144 raft_consensus.cc:738] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.018836 22144 consensus_queue.cc:260] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.034226 23307 leader_election.cc:290] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Requested vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.035552 23476 tablet_replica.cc:331] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
W20250112 02:57:41.039544 21383 catalog_manager.cc:6064] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: An error occurred while updating sys-tables: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER
I20250112 02:57:41.041414 23474 raft_consensus.cc:491] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.041880 23474 raft_consensus.cc:513] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.044123 23474 leader_election.cc:290] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.044714 23475 ts_tablet_manager.cc:1905] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.045773 22144 ts_tablet_manager.cc:1428] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.061s user 0.006s sys 0.001s
I20250112 02:57:41.047355 22144 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
W20250112 02:57:41.049404 21389 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 43) took 5884 ms (client timeout 1000 ms). Trace:
I20250112 02:57:41.050923 22145 tablet_bootstrap.cc:492] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:41.053347 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.055392 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet f445a4bf43db4571b6d3a4afe8fe823a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:24 UTC) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.055660 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9"
I20250112 02:57:41.057037 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2b248224f9416faf4bd83a853ec918 with delete_type TABLET_DATA_DELETED (replaced by tablet d1e0548276474b68919cb10fbcfcdf04) from {username='slave'} at 127.0.0.1:35272
I20250112 02:57:41.060032 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "779badaa22e6404aaaf1818e86228bd4" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:41.061167 21392 catalog_manager.cc:6064] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: An error occurred while updating sys-tables: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER
W20250112 02:57:41.073295 21836 consensus_peers.cc:487] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 -> Peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Couldn't send request to peer e0406c08e9714e87bdd7fd6b66c9756d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 247f0ef0ada44d9d957c8ff42b0da4b5. This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:41.076314 21980 leader_election.cc:343] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:41.076846 21980 leader_election.cc:343] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
W20250112 02:57:40.969244 21390 rpcz_store.cc:269] 0112 02:57:34.713391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:34.977532 (+264141us) service_pool.cc:225] Handling call
0112 02:57:36.117886 (+1140354us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.117908 (+ 22us) write_op.cc:454] Released schema lock
0112 02:57:40.968645 (+4850737us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:36.099811 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:36.100239 (+ 428us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:36.100264 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:36.100279 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:36.100805 (+ 526us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:36.100846 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:36.100869 (+ 23us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:36.101134 (+ 265us) tablet.cc:698] Row locks acquired
0112 02:57:36.101158 (+ 24us) write_op.cc:260] PREPARE: finished
0112 02:57:36.101266 (+ 108us) write_op.cc:270] Start()
0112 02:57:36.101346 (+ 80us) write_op.cc:276] Timestamp: P: 1736650656101381 usec, L: 0
0112 02:57:36.101364 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:36.101836 (+ 472us) log.cc:844] Serialized 1302 byte log entry
0112 02:57:36.111311 (+ 9475us) op_driver.cc:464] REPLICATION: finished
0112 02:57:36.111621 (+ 310us) write_op.cc:301] APPLY: starting
0112 02:57:36.111669 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:36.111779 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:36.111879 (+ 100us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:36.112177 (+ 298us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:36.112322 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:36.112358 (+ 36us) write_op.cc:312] APPLY: finished
0112 02:57:36.112531 (+ 173us) log.cc:844] Serialized 35 byte log entry
0112 02:57:36.112648 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.112717 (+ 69us) write_op.cc:454] Released schema lock
0112 02:57:36.112750 (+ 33us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":92435,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":192,"apply.run_cpu_time_us":1276,"apply.run_wall_time_us":1275,"num_ops":3,"prepare.queue_time_us":82,"prepare.run_cpu_time_us":2275,"prepare.run_wall_time_us":2274,"raft.queue_time_us":1248,"raft.run_cpu_time_us":562,"raft.run_wall_time_us":562,"replication_time_us":9863}]]}
W20250112 02:57:41.102679 21388 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.130:58729 (request call id 45) took 4723 ms (client timeout 1000 ms). Trace:
I20250112 02:57:41.103092 23334 tablet_loader.cc:96] loaded metadata for tablet 030d48c4421e45f0900705c4b6a4f4ed (table default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c])
W20250112 02:57:41.103092 21388 rpcz_store.cc:269] 0112 02:57:36.379461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.379685 (+ 224us) service_pool.cc:225] Handling call
0112 02:57:40.945361 (+4565676us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.945381 (+ 20us) write_op.cc:454] Released schema lock
0112 02:57:41.102611 (+157230us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:40.945134 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:40.945181 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.945207 (+ 26us) write_op.cc:454] Released schema lock
0112 02:57:40.945231 (+ 24us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":54788,"reported_tablets":37,"child_traces":[["op",{}]]}
I20250112 02:57:41.105082 23334 tablet_loader.cc:96] loaded metadata for tablet 05ba915163894cc6aa0cfb144dd49d97 (table default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd])
I20250112 02:57:41.108914 23334 tablet_loader.cc:96] loaded metadata for tablet 08e25cf536254f3ea9d2c1e97a32b85b (table default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b])
I20250112 02:57:41.110575 23334 tablet_loader.cc:96] loaded metadata for tablet 0df734ba87264abd803237a5bbb32da2 (table default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169])
I20250112 02:57:41.111547 23334 tablet_loader.cc:96] loaded metadata for tablet 118ef6d7c16941598177b92a94c4b472 (table default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43])
I20250112 02:57:41.114135 23334 tablet_loader.cc:96] loaded metadata for tablet 13edf2c56e8e4254b75e0eb85fb88427 (table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963])
I20250112 02:57:41.116003 23334 tablet_loader.cc:96] loaded metadata for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9])
W20250112 02:57:41.112773 21387 catalog_manager.cc:6064] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd: An error occurred while updating sys-tables: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER
I20250112 02:57:41.117149 23334 tablet_loader.cc:96] loaded metadata for tablet 1bbce2d946cd47398a3d0a1f72ca1678 (table default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158])
I20250112 02:57:41.118150 23334 tablet_loader.cc:96] loaded metadata for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e])
I20250112 02:57:41.119098 23334 tablet_loader.cc:96] loaded metadata for tablet 1cacc670998c4452bd3c03d99a62ad82 (table default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268])
I20250112 02:57:41.120014 23334 tablet_loader.cc:96] loaded metadata for tablet 1db9ff39ff2341199549155a2db0aaba (table default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043])
I20250112 02:57:41.121706 23334 tablet_loader.cc:96] loaded metadata for tablet 1f21e3c769364b58bd371aac338db1f2 (table default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f])
I20250112 02:57:41.125646 23334 tablet_loader.cc:96] loaded metadata for tablet 21dc0755b38746a39bd773780e48cda5 (table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682])
I20250112 02:57:41.126824 23334 tablet_loader.cc:96] loaded metadata for tablet 22aac4332d0e476d9758004972ef8ab2 (table default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330])
W20250112 02:57:41.049829 21389 rpcz_store.cc:269] 0112 02:57:35.165028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.224962 (+ 59934us) service_pool.cc:225] Handling call
0112 02:57:36.446928 (+1221966us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.446954 (+ 26us) write_op.cc:454] Released schema lock
0112 02:57:40.950659 (+4503705us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.950679 (+ 20us) write_op.cc:454] Released schema lock
0112 02:57:41.049341 (+ 98662us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:36.408297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:36.408712 (+ 415us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:36.408739 (+ 27us) write_op.cc:435] Acquired schema lock
0112 02:57:36.408757 (+ 18us) tablet.cc:659] Decoding operations
0112 02:57:36.409065 (+ 308us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:36.409125 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:36.409149 (+ 24us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:36.409312 (+ 163us) tablet.cc:698] Row locks acquired
0112 02:57:36.409325 (+ 13us) write_op.cc:260] PREPARE: finished
0112 02:57:36.409441 (+ 116us) write_op.cc:270] Start()
0112 02:57:36.409528 (+ 87us) write_op.cc:276] Timestamp: P: 1736650656409475 usec, L: 0
0112 02:57:36.409546 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:57:36.410070 (+ 524us) log.cc:844] Serialized 356 byte log entry
0112 02:57:36.419305 (+ 9235us) op_driver.cc:464] REPLICATION: finished
0112 02:57:36.419746 (+ 441us) write_op.cc:301] APPLY: starting
0112 02:57:36.419819 (+ 73us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:36.419935 (+ 116us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:36.420082 (+ 147us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:36.420236 (+ 154us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:36.420392 (+ 156us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:36.420453 (+ 61us) write_op.cc:312] APPLY: finished
0112 02:57:36.420650 (+ 197us) log.cc:844] Serialized 23 byte log entry
0112 02:57:36.420828 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:36.420910 (+ 82us) write_op.cc:454] Released schema lock
0112 02:57:36.420948 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:40.950453 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:40.950488 (+ 35us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.950541 (+ 53us) write_op.cc:454] Released schema lock
0112 02:57:40.950558 (+ 17us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":179508,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":278,"apply.run_cpu_time_us":1421,"apply.run_wall_time_us":1421,"num_ops":1,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":2049,"prepare.run_wall_time_us":2048,"raft.queue_time_us":1094,"raft.run_cpu_time_us":582,"raft.run_wall_time_us":582,"replication_time_us":9644}],["op",{}]]}
I20250112 02:57:41.129622 23334 tablet_loader.cc:96] loaded metadata for tablet 2722e4001ba44a7998f4ba7682bd3c44 (table default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd])
I20250112 02:57:41.132723 23334 tablet_loader.cc:96] loaded metadata for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb (table default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c])
I20250112 02:57:41.133694 23334 tablet_loader.cc:96] loaded metadata for tablet 2d793344526a404e8a737e7fb628e521 (table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec])
I20250112 02:57:41.134802 23334 tablet_loader.cc:96] loaded metadata for tablet 2f6c28220edf4598af19ba21e3e7b7d3 (table default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481])
I20250112 02:57:41.136612 23334 tablet_loader.cc:96] loaded metadata for tablet 334ed386f72d4fca870ade9d6a44589a (table default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9])
I20250112 02:57:41.141502 23334 tablet_loader.cc:96] loaded metadata for tablet 368af02998984cb2bee487b15b6a2b35 (table default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815])
I20250112 02:57:41.145422 23334 tablet_loader.cc:96] loaded metadata for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8 (table default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0])
I20250112 02:57:41.146485 23334 tablet_loader.cc:96] loaded metadata for tablet 3fef7ac20f034f589e564e5f5e754705 (table default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f])
I20250112 02:57:41.148322 23334 tablet_loader.cc:96] loaded metadata for tablet 41369773a0034c65b25fee871e6a25aa (table default.Table_03e04d34d58e463e93d16e858752d3cb [id=30ad19fdfc1246ebbeaf73fff91f925e])
I20250112 02:57:41.149883 23334 tablet_loader.cc:96] loaded metadata for tablet 43c413e1f8fe48039c4dd1b066fdd857 (table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143])
I20250112 02:57:41.151698 23334 tablet_loader.cc:96] loaded metadata for tablet 4494e809c9ee4a0884da38b45e7a90b2 (table default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152])
I20250112 02:57:41.153427 23334 tablet_loader.cc:96] loaded metadata for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d])
I20250112 02:57:41.154541 23334 tablet_loader.cc:96] loaded metadata for tablet 486dd5218bad42b0bb9064b7a14f3a4e (table default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac])
I20250112 02:57:41.155599 23334 tablet_loader.cc:96] loaded metadata for tablet 4ce1f4a9244c4b40ad8557de5b36a40e (table default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf])
I20250112 02:57:41.157614 23334 tablet_loader.cc:96] loaded metadata for tablet 4e4262c2fdc0440ea696e551d00b80e9 (table default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2])
I20250112 02:57:41.158617 22144 tablet_bootstrap.cc:654] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.158761 23334 tablet_loader.cc:96] loaded metadata for tablet 557de3de9a4d4d0bac1dc16f66ab6b01 (table default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3])
I20250112 02:57:41.159795 23334 tablet_loader.cc:96] loaded metadata for tablet 56c7f07da89c4994a57c23ebdd8061bf (table default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2])
I20250112 02:57:41.161566 23334 tablet_loader.cc:96] loaded metadata for tablet 58c49391d2ed4c3fa84fa2d071b06094 (table default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59])
I20250112 02:57:41.163333 23334 tablet_loader.cc:96] loaded metadata for tablet 5c3fd2d14dbf4bc98e42a04e33c66348 (table default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3])
I20250112 02:57:41.164417 23334 tablet_loader.cc:96] loaded metadata for tablet 5d017c05abe94657bfbd2b38329bba16 (table default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f])
I20250112 02:57:41.168195 23334 tablet_loader.cc:96] loaded metadata for tablet 62253be16a1a4b2cb7d50264e82478fc (table default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12])
I20250112 02:57:41.169171 23476 raft_consensus.cc:2238] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.169701 23476 raft_consensus.cc:2267] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 02:57:41.169718 21392 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 66fe67a814504a89b6fa8f46e2a4aee9 consensus state has no leader
I20250112 02:57:41.170962 23334 tablet_loader.cc:96] loaded metadata for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61])
I20250112 02:57:41.171548 23476 ts_tablet_manager.cc:1905] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.171953 23334 tablet_loader.cc:96] loaded metadata for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575])
I20250112 02:57:41.172048 22145 tablet_bootstrap.cc:654] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.173060 23334 tablet_loader.cc:96] loaded metadata for tablet 67236f9e5d11467486e81393dd01e0df (table default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e])
I20250112 02:57:41.174172 23334 tablet_loader.cc:96] loaded metadata for tablet 67260685978247898ad224128ed8964f (table default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb])
I20250112 02:57:41.175357 23334 tablet_loader.cc:96] loaded metadata for tablet 685953572d45476ebaf2580fd5e4a3cd (table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8])
I20250112 02:57:41.176671 23334 tablet_loader.cc:96] loaded metadata for tablet 68f5144c9c644a7f8ca95082459c4556 (table default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e])
I20250112 02:57:41.177745 23334 tablet_loader.cc:96] loaded metadata for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597])
I20250112 02:57:41.180461 23334 tablet_loader.cc:96] loaded metadata for tablet 6e922b68d33b4a76a260816475ee6a3c (table default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d])
I20250112 02:57:41.183226 23307 raft_consensus.cc:491] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.183638 23307 raft_consensus.cc:513] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.185431 23307 leader_election.cc:290] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.187397 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a08141e3615244a9b266cd197d9b9ef3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:41.188681 23477 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:41.188910 23334 tablet_loader.cc:96] loaded metadata for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740])
I20250112 02:57:41.189307 23477 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
W20250112 02:57:41.190799 21980 leader_election.cc:343] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.198679 23334 tablet_loader.cc:96] loaded metadata for tablet 7634a1d8de9b495599b83f15eaa91c89 (table default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0])
I20250112 02:57:41.199790 23334 tablet_loader.cc:96] loaded metadata for tablet 779badaa22e6404aaaf1818e86228bd4 (table default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c])
I20250112 02:57:41.205183 22144 tablet_bootstrap.cc:492] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:41.205575 22144 ts_tablet_manager.cc:1397] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.003s
I20250112 02:57:41.207859 22144 raft_consensus.cc:357] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.208451 22144 raft_consensus.cc:383] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.208707 22144 raft_consensus.cc:738] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.213238 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet be5f2a4f6a8e4a64a7642d5a18b58a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.217514 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet a4b543ed18b34cd58a1c588ed7d079c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 95f62433f15a43eb9a654c067052cd68) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.217353 22144 consensus_queue.cc:260] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
E20250112 02:57:40.964013 21386 catalog_manager.cc:5681] Error updating tablets from e0406c08e9714e87bdd7fd6b66c9756d: Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "edfa495bfefc46b998c2f870d9014900" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "779badaa22e6404aaaf1818e86228bd4" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "750246f0a0c549229d4e7a87cd6ddfd9" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "334ed386f72d4fca870ade9d6a44589a" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "030d48c4421e45f0900705c4b6a4f4ed" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "685953572d45476ebaf2580fd5e4a3cd" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "ae3ed0bdc2ec44998ee461d4b1254e91" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2d793344526a404e8a737e7fb628e521" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "78f03a1dfb604b33a0528c28ade287f2" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "66fe67a814504a89b6fa8f46e2a4aee9" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "4716677cfad74c0dafe18c55796aeae5" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "b0d0ba05ee4c4c9f815a4c04830b5dd4" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e4a9bc28dc5942608fb7224f9f058e27" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "82825c2061d74cafa814912bec005654" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "46f8052651fa496cbb3702c17f592106" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e21a0e4b5ebc426383cf332f46492b80" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e7d8db14a8054ae69daf42dd76f35511" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "68f5144c9c644a7f8ca95082459c4556" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0df734ba87264abd803237a5bbb32da2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "e0977a10252840a381234f02bb7339b3" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "67260685978247898ad224128ed8964f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a69123c4ab384e0380c62e52b9f3b42d" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8b6efb19f0cc4a96b957ec34c27ab836" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2722e4001ba44a7998f4ba7682bd3c44" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "d6ca275360ec41ceb0f7ef9bfaedf647" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1f21e3c769364b58bd371aac338db1f2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "cca160b7cffb4ae287ea077e7352ffc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "cf8bf172abad43ac9a519310334d1640" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1c20f2a3dfbc4de2bde9491e17d6ba9a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27069 live_row_count: 0 } } updated_tablets { tablet_id: "eef30508946e4376961b6cd463172c1f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "65eb8c99917247edbdbf5de42f3b505c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } updated_tablets { tablet_id: "f47e14720a704a19ac3aea8723e13c28" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26847 live_row_count: 0 } } updated_tablets { tablet_id: "922b22f0b1284bc4a732371a61cb1ed4" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f780cf38ac4d41d8b59a3d8882a10030" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "22aac4332d0e476d9758004972ef8ab2" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "118ef6d7c16941598177b92a94c4b472" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 26772 live_row_count: 0 } } updated_tablets { tablet_id: "f3da552a973945e284a386304f182777" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6e922b68d33b4a76a260816475ee6a3c" state: RUNNING consensus_state { current_term: 1 leader_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "2cbf5e2a26474c98b9cb48b5bbdceafb" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "1800d0eb7a9a48d295aca16607291add" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7c9b3e6c84484575acd46e61dd1ea28a" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f08c1affbaf247059ab65222e2e95bab" state: RUNNING consensus_state { current_term: 1 leader_uuid: "1c2770701ca54288956df9d910d7d3d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } } pe
I20250112 02:57:41.221963 23476 ts_tablet_manager.cc:1918] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:41.222307 23476 log.cc:1198] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/46f8052651fa496cbb3702c17f592106
I20250112 02:57:41.216427 23334 tablet_loader.cc:96] loaded metadata for tablet 78f03a1dfb604b33a0528c28ade287f2 (table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a])
I20250112 02:57:41.230305 23334 tablet_loader.cc:96] loaded metadata for tablet 7a1144ac759a4db496f67cdc9ec35020 (table default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7])
I20250112 02:57:41.231321 23334 tablet_loader.cc:96] loaded metadata for tablet 7af7e02e5fc248839b434f42f95c6509 (table default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48])
I20250112 02:57:41.232306 23334 tablet_loader.cc:96] loaded metadata for tablet 7c9b3e6c84484575acd46e61dd1ea28a (table default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae])
I20250112 02:57:41.235850 23477 leader_election.cc:290] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9ef739ef9dc460ca8287b1d2c6c2217 (127.20.217.189:36049), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:41.235760 22878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 7 candidate_status { last_received { term: 6 index: 367 } } ignore_live_leader: false dest_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" is_pre_election: true
I20250112 02:57:41.233656 22019 tablet_service.cc:1514] Processing DeleteTablet for tablet f3da552a973945e284a386304f182777 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:31 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.234076 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 922b22f0b1284bc4a732371a61cb1ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:32 UTC) from {username='slave'} at 127.0.0.1:53076
W20250112 02:57:41.249572 22841 leader_election.cc:336] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683): Timed out: RequestConsensusVote RPC to 127.20.217.188:35683 timed out after 0.612s (SENT)
I20250112 02:57:41.254877 23476 ts_tablet_manager.cc:1939] T 46f8052651fa496cbb3702c17f592106 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:41.262138 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 2601daad8ba742ec88ccfe603440e8e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:27 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.264506 23334 tablet_loader.cc:96] loaded metadata for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633])
I20250112 02:57:41.265569 23334 tablet_loader.cc:96] loaded metadata for tablet 82d2ac91c5554a64936f364e4e1978f8 (table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab])
I20250112 02:57:41.267242 22144 ts_tablet_manager.cc:1428] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.061s user 0.006s sys 0.001s
I20250112 02:57:41.267313 22145 tablet_bootstrap.cc:492] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:41.267374 23334 tablet_loader.cc:96] loaded metadata for tablet 8a8d7d76e5184d668e4add9810fab7df (table default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f])
I20250112 02:57:41.267645 22145 ts_tablet_manager.cc:1397] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.217s user 0.012s sys 0.002s
I20250112 02:57:41.268947 23334 tablet_loader.cc:96] loaded metadata for tablet 8b39a5b05b0941aea541a250bb64941e (table default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4])
I20250112 02:57:41.269529 22145 raft_consensus.cc:357] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.270102 22145 raft_consensus.cc:383] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:41.280179 21392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.129:43111 (request call id 46) took 4818 ms (client timeout 1000 ms). Trace:
W20250112 02:57:41.280479 21392 rpcz_store.cc:269] 0112 02:57:36.462010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.462139 (+ 129us) service_pool.cc:225] Handling call
0112 02:57:41.060850 (+4598711us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.060869 (+ 19us) write_op.cc:454] Released schema lock
0112 02:57:41.280115 (+219246us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:41.060657 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:41.060691 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.060711 (+ 20us) write_op.cc:454] Released schema lock
0112 02:57:41.060729 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":77899,"reported_tablets":47,"child_traces":[["op",{}]]}
I20250112 02:57:41.283447 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:26 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.284775 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-12 02:57:35 UTC) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.270488 23334 tablet_loader.cc:96] loaded metadata for tablet 8b6efb19f0cc4a96b957ec34c27ab836 (table default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7])
I20250112 02:57:41.270314 22145 raft_consensus.cc:738] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.290782 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]) successfully deleted
I20250112 02:57:41.293365 23334 tablet_loader.cc:96] loaded metadata for tablet 929430298434492cbab3d197e1083400 (table default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde])
I20250112 02:57:41.295181 23334 tablet_loader.cc:96] loaded metadata for tablet 95f62433f15a43eb9a654c067052cd68 (table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb])
I20250112 02:57:41.300845 22144 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:41.294139 22145 consensus_queue.cc:260] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.291067 23476 tablet_replica.cc:331] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:41.309566 22015 tablet_service.cc:1514] Processing DeleteTablet for tablet 443aa544f6df4d50a78c9effd306e216 with delete_type TABLET_DATA_DELETED (replaced by tablet b9034065c6bc435d9f26f85a96297cd3) from {username='slave'} at 127.0.0.1:53076
I20250112 02:57:41.312852 23334 tablet_loader.cc:96] loaded metadata for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4])
W20250112 02:57:41.313081 21386 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 41) took 3995 ms (client timeout 1000 ms). Trace:
W20250112 02:57:41.313362 21386 rpcz_store.cc:269] 0112 02:57:37.317666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:37.317801 (+ 135us) service_pool.cc:225] Handling call
0112 02:57:40.963799 (+3645998us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.963818 (+ 19us) write_op.cc:454] Released schema lock
0112 02:57:41.313006 (+349188us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0112 02:57:40.963619 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:40.963653 (+ 34us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:40.963678 (+ 25us) write_op.cc:454] Released schema lock
0112 02:57:40.963703 (+ 25us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":20610,"reported_tablets":60,"child_traces":[["op",{}]]}
I20250112 02:57:41.314610 23334 tablet_loader.cc:96] loaded metadata for tablet a08141e3615244a9b266cd197d9b9ef3 (table default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4])
I20250112 02:57:41.316323 23334 tablet_loader.cc:96] loaded metadata for tablet a345af15e7664d01a184193ddb51572b (table default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db])
I20250112 02:57:41.319119 23475 ts_tablet_manager.cc:1918] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250112 02:57:41.319562 23475 log.cc:1198] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/46f8052651fa496cbb3702c17f592106
I20250112 02:57:41.320431 23475 ts_tablet_manager.cc:1939] T 46f8052651fa496cbb3702c17f592106 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
W20250112 02:57:41.322046 21387 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 39) took 5258 ms (client timeout 1000 ms). Trace:
W20250112 02:57:41.324071 21383 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.20.217.131:60225 (request call id 38) took 6520 ms (client timeout 1000 ms). Trace:
W20250112 02:57:41.324366 21383 rpcz_store.cc:269] 0112 02:57:34.803315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:35.099097 (+295782us) service_pool.cc:225] Handling call
0112 02:57:41.039266 (+5940169us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.039283 (+ 17us) write_op.cc:454] Released schema lock
0112 02:57:41.191592 (+152309us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b680003e960. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eef2fb931 00007f7eef2d919a 00007f7eef2d78d8 00007f7eef2d5ee0 00007f7eef3d76ae 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc
0112 02:57:41.324013 (+132421us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:41.039105 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:41.039135 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.039152 (+ 17us) write_op.cc:454] Released schema lock
0112 02:57:41.039170 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":118082,"reported_tablets":51,"spinlock_wait_cycles":66205952,"child_traces":[["op",{}]]}
I20250112 02:57:41.328013 23334 tablet_loader.cc:96] loaded metadata for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_181a7bdabe584055b44baf7ff5ab440f [id=8dbf245ab5354db7a93b54749e978c65])
I20250112 02:57:41.317764 22145 ts_tablet_manager.cc:1428] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.050s user 0.007s sys 0.000s
I20250112 02:57:41.329759 23334 tablet_loader.cc:96] loaded metadata for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748])
I20250112 02:57:41.330821 23334 tablet_loader.cc:96] loaded metadata for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c (table default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680])
I20250112 02:57:41.331835 23334 tablet_loader.cc:96] loaded metadata for tablet ae3ed0bdc2ec44998ee461d4b1254e91 (table default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2])
I20250112 02:57:41.333472 23334 tablet_loader.cc:96] loaded metadata for tablet af0c74dd3303445b9e9aef1f2d7af62a (table default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a])
I20250112 02:57:41.336632 22145 tablet_bootstrap.cc:492] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
W20250112 02:57:41.322319 21387 rpcz_store.cc:269] 0112 02:57:36.064071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:36.064224 (+ 153us) service_pool.cc:225] Handling call
0112 02:57:40.858648 (+4794424us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b680003dd60. stack: 00007f7ee66b6853 00007f7ee66b6750 00007f7ee622b584 00000000004fcd24 00000000004fccb9 00007f7eef2fb931 00007f7eef2d919a 00007f7eef2d78d8 00007f7eef2d5ee0 00007f7eef3d76ae 00007f7eed134fc4 00007f7eed134f52 00007f7eed134e81 00007f7eed134dfc 00007f7eed1340b2 00007f7eeb3fedfc
0112 02:57:41.094106 (+235458us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.095137 (+ 1031us) write_op.cc:454] Released schema lock
0112 02:57:41.321990 (+226853us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:41.080092 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica bcc0757376c3411a9fcbc84ec411befd is not leader of this config: current role FOLLOWER)
0112 02:57:41.080963 (+ 871us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:41.082119 (+ 1156us) write_op.cc:454] Released schema lock
0112 02:57:41.082812 (+ 693us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":119118,"reported_tablets":57,"spinlock_wait_cycles":4203776,"child_traces":[["op",{}]]}
I20250112 02:57:41.349404 22144 tablet_bootstrap.cc:654] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.351339 23474 raft_consensus.cc:491] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.351796 23474 raft_consensus.cc:513] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.354250 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbce2d946cd47398a3d0a1f72ca1678" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:41.355435 21980 leader_election.cc:343] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.355657 23474 leader_election.cc:290] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.368162 23334 tablet_loader.cc:96] loaded metadata for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d])
I20250112 02:57:41.370211 23476 raft_consensus.cc:2238] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.370733 23476 raft_consensus.cc:2267] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.373162 23476 ts_tablet_manager.cc:1905] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.392621 23474 raft_consensus.cc:491] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.393079 23474 raft_consensus.cc:513] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.395021 23474 leader_election.cc:290] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.396183 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b32e6564d4dd4d1b9b25a14266b3dc04" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:41.397279 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 46f8052651fa496cbb3702c17f592106 (table default.Table_45510e3d77df4fa899940125a4a5da9c [id=fd914cd362c9443c943ec3dd8a1de186]) successfully deleted
W20250112 02:57:41.397943 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 994e1d114e5a4de5aa1d7651907d718e because the tablet was not found. No further retry: Not found: Tablet not found: 994e1d114e5a4de5aa1d7651907d718e
W20250112 02:57:41.398403 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 5e3890251e644c1eb596de043b63491e because the tablet was not found. No further retry: Not found: Tablet not found: 5e3890251e644c1eb596de043b63491e
I20250112 02:57:41.398677 23334 tablet_loader.cc:96] loaded metadata for tablet b32e6564d4dd4d1b9b25a14266b3dc04 (table default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b])
I20250112 02:57:41.398558 23475 tablet_replica.cc:331] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
W20250112 02:57:41.398883 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 96f98f6840e24f3c9d03937c1bbc00e7 because the tablet was not found. No further retry: Not found: Tablet not found: 96f98f6840e24f3c9d03937c1bbc00e7
W20250112 02:57:41.399322 21378 catalog_manager.cc:4839] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): delete failed for tablet 84f7700b8cce4261a32f2009d1481b99 because the tablet was not found. No further retry: Not found: Tablet not found: 84f7700b8cce4261a32f2009d1481b99
I20250112 02:57:41.399825 23334 tablet_loader.cc:96] loaded metadata for tablet b4aff0623f504bf09011ca6de28a3b82 (table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd])
W20250112 02:57:41.399920 21980 leader_election.cc:343] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.400242 23475 raft_consensus.cc:2238] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.400861 23334 tablet_loader.cc:96] loaded metadata for tablet b4dddfea8e8b49189186930fce46ecc5 (table default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0])
I20250112 02:57:41.401129 23475 raft_consensus.cc:2267] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.401906 23334 tablet_loader.cc:96] loaded metadata for tablet b9034065c6bc435d9f26f85a96297cd3 (table default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5])
I20250112 02:57:41.404830 23475 ts_tablet_manager.cc:1905] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.436899 22144 tablet_bootstrap.cc:492] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:41.437283 22144 ts_tablet_manager.cc:1397] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.004s
I20250112 02:57:41.439396 22144 raft_consensus.cc:357] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.439980 22144 raft_consensus.cc:383] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.440220 22144 raft_consensus.cc:738] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.440753 22144 consensus_queue.cc:260] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.443955 23334 tablet_loader.cc:96] loaded metadata for tablet be0b4a02597842d3ae1ad0c6197cc808 (table default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041])
I20250112 02:57:41.445837 23334 tablet_loader.cc:96] loaded metadata for tablet c01590521b6e4ddc9d7fcc7142d49930 (table default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5])
I20250112 02:57:41.448163 22145 tablet_bootstrap.cc:654] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.475762 23334 tablet_loader.cc:96] loaded metadata for tablet c80b7813b50d4739a76776d5d16434f3 (table default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779])
I20250112 02:57:41.477545 23334 tablet_loader.cc:96] loaded metadata for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a])
I20250112 02:57:41.478906 22144 ts_tablet_manager.cc:1428] T e7d8db14a8054ae69daf42dd76f35511 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250112 02:57:41.479784 22144 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:41.482569 23360 raft_consensus.cc:491] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.482983 23360 raft_consensus.cc:513] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.484970 23360 leader_election.cc:290] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:41.486565 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13edf2c56e8e4254b75e0eb85fb88427" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:41.492293 23481 raft_consensus.cc:491] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e76221f7d4ec44da987303d68c393a1b)
I20250112 02:57:41.500427 23476 ts_tablet_manager.cc:1918] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:41.512903 22144 tablet_bootstrap.cc:654] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.515180 23334 tablet_loader.cc:96] loaded metadata for tablet d1e0548276474b68919cb10fbcfcdf04 (table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618])
I20250112 02:57:41.501849 23481 raft_consensus.cc:513] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcc0757376c3411a9fcbc84ec411befd" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 34893 } } peers { permanent_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 36049 } } peers { permanent_uuid: "e76221f7d4ec44da987303d68c393a1b" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 35683 } }
I20250112 02:57:41.517868 23334 tablet_loader.cc:96] loaded metadata for tablet d6ca275360ec41ceb0f7ef9bfaedf647 (table default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f])
I20250112 02:57:41.527127 23334 tablet_loader.cc:96] loaded metadata for tablet d94cda34ff2e44cfa543362a007f1400 (table default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86])
I20250112 02:57:41.528337 23334 tablet_loader.cc:96] loaded metadata for tablet da54d9dbab2f416785ec0379095bcf3e (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96])
I20250112 02:57:41.529160 21401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 7 candidate_status { last_received { term: 6 index: 367 } } ignore_live_leader: false dest_uuid: "bcc0757376c3411a9fcbc84ec411befd" is_pre_election: true
I20250112 02:57:41.529552 23334 tablet_loader.cc:96] loaded metadata for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064])
I20250112 02:57:41.530779 23334 tablet_loader.cc:96] loaded metadata for tablet e0e6beaabb6d4b099ce9e47736bb29b4 (table default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c])
I20250112 02:57:41.525550 23474 raft_consensus.cc:491] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.532395 23474 raft_consensus.cc:513] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.544809 23476 log.cc:1198] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/be5f2a4f6a8e4a64a7642d5a18b58a6d
W20250112 02:57:41.544960 21836 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.131:39307, user_credentials={real_user=slave}} blocked reactor thread for 50131.1us
I20250112 02:57:41.554837 23476 ts_tablet_manager.cc:1939] T be5f2a4f6a8e4a64a7642d5a18b58a6d P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:41.558555 22144 tablet_bootstrap.cc:492] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:41.560137 23474 leader_election.cc:290] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.562649 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c3fd2d14dbf4bc98e42a04e33c66348" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:41.563038 22144 ts_tablet_manager.cc:1397] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.000s
I20250112 02:57:41.563530 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet be5f2a4f6a8e4a64a7642d5a18b58a6d (table default.Table_60849e78a19643198da449366a8e6ed6 [id=7b2020d7f2ae4cae91616b74c1410aa4]) successfully deleted
W20250112 02:57:41.564204 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet a4b543ed18b34cd58a1c588ed7d079c5 because the tablet was not found. No further retry: Not found: Tablet not found: a4b543ed18b34cd58a1c588ed7d079c5
I20250112 02:57:41.565205 22144 raft_consensus.cc:357] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.565860 22144 raft_consensus.cc:383] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.566103 22144 raft_consensus.cc:738] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
W20250112 02:57:41.565429 21980 leader_election.cc:343] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.562405 23476 tablet_replica.cc:331] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:41.575587 22145 tablet_bootstrap.cc:492] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:41.576225 22145 ts_tablet_manager.cc:1397] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.242s user 0.020s sys 0.003s
I20250112 02:57:41.574440 23481 leader_election.cc:290] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcc0757376c3411a9fcbc84ec411befd (127.20.217.190:34893), e76221f7d4ec44da987303d68c393a1b (127.20.217.188:35683)
I20250112 02:57:41.586684 22144 consensus_queue.cc:260] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.588692 22144 ts_tablet_manager.cc:1428] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20250112 02:57:41.531952 23334 tablet_loader.cc:96] loaded metadata for tablet e21a0e4b5ebc426383cf332f46492b80 (table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce])
I20250112 02:57:41.600548 23334 tablet_loader.cc:96] loaded metadata for tablet e4a9bc28dc5942608fb7224f9f058e27 (table default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d])
I20250112 02:57:41.601624 23334 tablet_loader.cc:96] loaded metadata for tablet e7d8db14a8054ae69daf42dd76f35511 (table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718])
I20250112 02:57:41.618306 23334 tablet_loader.cc:96] loaded metadata for tablet e861bab8dcef41328095954a46e00a96 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669])
I20250112 02:57:41.619292 23474 raft_consensus.cc:491] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.619695 23474 raft_consensus.cc:513] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.621421 23474 leader_election.cc:290] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.622304 23334 tablet_loader.cc:96] loaded metadata for tablet e8f7301bed6c4d98a80c12b70073e62f (table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d])
I20250112 02:57:41.622625 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08e25cf536254f3ea9d2c1e97a32b85b" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:41.623402 23334 tablet_loader.cc:96] loaded metadata for tablet eab717cb5d794d15b9b51881b86501fc (table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac])
W20250112 02:57:41.623914 21980 leader_election.cc:343] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.625223 23334 tablet_loader.cc:96] loaded metadata for tablet ee7967f6294b4cfeac9e97db8de41e11 (table default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0])
I20250112 02:57:41.626307 23334 tablet_loader.cc:96] loaded metadata for tablet eef30508946e4376961b6cd463172c1f (table default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5])
I20250112 02:57:41.627343 23334 tablet_loader.cc:96] loaded metadata for tablet f08c1affbaf247059ab65222e2e95bab (table default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d])
I20250112 02:57:41.628408 23334 tablet_loader.cc:96] loaded metadata for tablet f226759c7d8b4b0d83395c542428f4c9 (table default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18])
I20250112 02:57:41.630901 23334 tablet_loader.cc:96] loaded metadata for tablet f47e14720a704a19ac3aea8723e13c28 (table default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2])
I20250112 02:57:41.632443 23334 tablet_loader.cc:96] loaded metadata for tablet f663e183c2ed49b98327b2ad0de12ed9 (table default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952])
I20250112 02:57:41.632525 22145 raft_consensus.cc:357] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.633342 22145 raft_consensus.cc:383] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.633569 23334 tablet_loader.cc:96] loaded metadata for tablet f780cf38ac4d41d8b59a3d8882a10030 (table default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822])
I20250112 02:57:41.633651 22145 raft_consensus.cc:738] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.634351 22145 consensus_queue.cc:260] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.658705 23482 raft_consensus.cc:491] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.659188 23482 raft_consensus.cc:513] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.661949 23482 leader_election.cc:290] T be0b4a02597842d3ae1ad0c6197cc808 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:41.663200 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0b4a02597842d3ae1ad0c6197cc808" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:41.669211 22144 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:41.674430 22144 tablet_bootstrap.cc:654] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.690455 22145 ts_tablet_manager.cc:1428] T ee7967f6294b4cfeac9e97db8de41e11 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.114s user 0.005s sys 0.000s
I20250112 02:57:41.692488 22145 tablet_bootstrap.cc:492] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:41.711417 21550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 7 candidate_status { last_received { term: 6 index: 367 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:41.711601 21556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b"
W20250112 02:57:41.712239 21556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40502 (request call id 562) took 2355 ms (client timeout 545 ms). Trace:
W20250112 02:57:41.712173 21550 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40502 (request call id 564) took 475 ms (client timeout 592 ms). Trace:
W20250112 02:57:41.712532 21556 rpcz_store.cc:269] 0112 02:57:39.357313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:39.357484 (+ 171us) service_pool.cc:225] Handling call
0112 02:57:41.712173 (+2354689us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 02:57:41.712572 21550 rpcz_store.cc:269] 0112 02:57:41.236372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:41.236526 (+ 154us) service_pool.cc:225] Handling call
0112 02:57:41.712107 (+475581us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:41.713433 21552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 5 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:41.711441 21551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 5 candidate_status { last_received { term: 4 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
W20250112 02:57:41.714200 21552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34308 (request call id 6) took 1771 ms (client timeout 612 ms). Trace:
W20250112 02:57:41.714293 21551 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34308 (request call id 7) took 1077 ms (client timeout 612 ms). Trace:
W20250112 02:57:41.714527 21552 rpcz_store.cc:269] 0112 02:57:39.943271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:39.943523 (+ 252us) service_pool.cc:225] Handling call
0112 02:57:41.714128 (+1770605us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 02:57:41.714654 21551 rpcz_store.cc:269] 0112 02:57:40.636806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:40.636992 (+ 186us) service_pool.cc:225] Handling call
0112 02:57:41.714234 (+1077242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:41.715457 21554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:41.715893 21553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
W20250112 02:57:41.716115 21554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40502 (request call id 560) took 2985 ms (client timeout 716 ms). Trace:
W20250112 02:57:41.716574 21553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40502 (request call id 563) took 2270 ms (client timeout 716 ms). Trace:
W20250112 02:57:41.716924 21553 rpcz_store.cc:269] 0112 02:57:39.446394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:39.446689 (+ 295us) service_pool.cc:225] Handling call
0112 02:57:41.716525 (+2269836us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:41.716960 21549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9ef739ef9dc460ca8287b1d2c6c2217" candidate_term: 7 candidate_status { last_received { term: 6 index: 367 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b" is_pre_election: true
I20250112 02:57:41.717599 21555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bcc0757376c3411a9fcbc84ec411befd" candidate_term: 5 candidate_status { last_received { term: 3 index: 366 } } ignore_live_leader: false dest_uuid: "e76221f7d4ec44da987303d68c393a1b"
I20250112 02:57:41.714336 21378 leader_election.cc:304] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcc0757376c3411a9fcbc84ec411befd; no voters: e76221f7d4ec44da987303d68c393a1b, e9ef739ef9dc460ca8287b1d2c6c2217
W20250112 02:57:41.718156 21555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40502 (request call id 561) took 2961 ms (client timeout 545 ms). Trace:
W20250112 02:57:41.718390 21555 rpcz_store.cc:269] 0112 02:57:38.756411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.756532 (+ 121us) service_pool.cc:225] Handling call
0112 02:57:41.718109 (+2961577us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250112 02:57:41.718457 22841 leader_election.cc:304] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9ef739ef9dc460ca8287b1d2c6c2217; no voters: bcc0757376c3411a9fcbc84ec411befd, e76221f7d4ec44da987303d68c393a1b
I20250112 02:57:41.719750 23477 raft_consensus.cc:2743] T 00000000000000000000000000000000 P bcc0757376c3411a9fcbc84ec411befd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250112 02:57:41.720288 23481 raft_consensus.cc:2743] T 00000000000000000000000000000000 P e9ef739ef9dc460ca8287b1d2c6c2217 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250112 02:57:41.716461 21554 rpcz_store.cc:269] 0112 02:57:38.730470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.730782 (+ 312us) service_pool.cc:225] Handling call
0112 02:57:41.716067 (+2985285us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 02:57:41.723094 23334 catalog_manager.cc:1477] Time spent T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Loading table and tablet metadata into memory: real 3.346s user 0.369s sys 0.030s
I20250112 02:57:41.730253 23334 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:57:41.752207 22145 tablet_bootstrap.cc:654] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.755751 23476 raft_consensus.cc:2238] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.756230 23476 raft_consensus.cc:2267] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.757884 23476 ts_tablet_manager.cc:1905] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.718351 23475 ts_tablet_manager.cc:1918] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:41.758772 23475 log.cc:1198] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/2601daad8ba742ec88ccfe603440e8e7
I20250112 02:57:41.759464 23475 ts_tablet_manager.cc:1939] T 2601daad8ba742ec88ccfe603440e8e7 P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:41.770643 22144 tablet_bootstrap.cc:492] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:41.771057 22144 ts_tablet_manager.cc:1397] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.102s user 0.015s sys 0.013s
I20250112 02:57:41.772989 23476 ts_tablet_manager.cc:1918] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:41.773337 23476 log.cc:1198] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/f3da552a973945e284a386304f182777
I20250112 02:57:41.773247 22144 raft_consensus.cc:357] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.774049 22144 raft_consensus.cc:383] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.774240 23476 ts_tablet_manager.cc:1939] T f3da552a973945e284a386304f182777 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:41.774292 22144 raft_consensus.cc:738] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.774976 22144 consensus_queue.cc:260] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.775447 23334 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:41.775687 23334 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:57:41.790380 23474 raft_consensus.cc:491] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.790809 23474 raft_consensus.cc:513] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.792429 23474 leader_election.cc:290] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.793689 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab717cb5d794d15b9b51881b86501fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:41.795387 21980 leader_election.cc:343] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.803202 22144 ts_tablet_manager.cc:1428] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250112 02:57:41.808737 23476 tablet_replica.cc:331] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: stopping tablet replica
I20250112 02:57:41.808987 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet f3da552a973945e284a386304f182777 (table default.Table_2573b35f681a4ede8201db20c64d179e [id=80273bec17084713b06f8f7f511fe6dd]) successfully deleted
I20250112 02:57:41.809608 23476 raft_consensus.cc:2238] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.810034 23476 raft_consensus.cc:2267] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.818755 23476 ts_tablet_manager.cc:1905] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.822950 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet 2601daad8ba742ec88ccfe603440e8e7 (table default.Table_33ea8d4190104eb8837824b516dd2314 [id=097357eb9ce548aa9fe2db1a906c70d8]) successfully deleted
I20250112 02:57:41.831053 22144 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:41.834743 23474 raft_consensus.cc:491] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.835140 23474 raft_consensus.cc:513] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.837414 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2021384bb41b58a66fcb7afaeaca8" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:41.838534 23474 leader_election.cc:290] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
W20250112 02:57:41.839141 21980 leader_election.cc:343] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.842676 23475 tablet_replica.cc:331] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:41.843641 23475 raft_consensus.cc:2238] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:41.844179 23475 raft_consensus.cc:2267] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:41.846113 23475 ts_tablet_manager.cc:1905] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:41.856678 23476 ts_tablet_manager.cc:1918] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:41.864362 23334 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:57:41.862182 22145 tablet_bootstrap.cc:492] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:41.875316 22145 ts_tablet_manager.cc:1397] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.183s user 0.003s sys 0.006s
I20250112 02:57:41.877346 22145 raft_consensus.cc:357] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.877979 22145 raft_consensus.cc:383] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.878211 22145 raft_consensus.cc:738] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.878934 22145 consensus_queue.cc:260] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.882951 23474 raft_consensus.cc:491] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.883365 23474 raft_consensus.cc:513] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.885142 23474 leader_election.cc:290] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:41.886514 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62253be16a1a4b2cb7d50264e82478fc" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
W20250112 02:57:41.887488 21980 leader_election.cc:343] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:41.898250 22145 ts_tablet_manager.cc:1428] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20250112 02:57:41.908696 23476 log.cc:1198] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/922b22f0b1284bc4a732371a61cb1ed4
I20250112 02:57:41.909469 23476 ts_tablet_manager.cc:1939] T 922b22f0b1284bc4a732371a61cb1ed4 P e0406c08e9714e87bdd7fd6b66c9756d: Deleting consensus metadata
I20250112 02:57:39.950902 22881 rpcz_store.cc:276] 0112 02:57:38.728276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:38.728438 (+ 162us) service_pool.cc:225] Handling call
0112 02:57:38.734136 (+ 5698us) raft_consensus.cc:1466] Updating replica for 60 ops
0112 02:57:38.781186 (+ 47050us) raft_consensus.cc:1529] Early marking committed up to index 0
0112 02:57:38.781213 (+ 27us) raft_consensus.cc:1534] Triggering prepare for 60 ops
0112 02:57:38.781600 (+ 387us) write_op.cc:270] Start()
0112 02:57:38.781690 (+ 90us) write_op.cc:276] Timestamp: P: 1736650649844367 usec, L: 0
0112 02:57:38.820357 (+ 38667us) write_op.cc:270] Start()
0112 02:57:38.820450 (+ 93us) write_op.cc:276] Timestamp: P: 1736650650317871 usec, L: 0
0112 02:57:38.821072 (+ 622us) write_op.cc:270] Start()
0112 02:57:38.821146 (+ 74us) write_op.cc:276] Timestamp: P: 1736650650727609 usec, L: 0
0112 02:57:38.821564 (+ 418us) write_op.cc:270] Start()
0112 02:57:38.821633 (+ 69us) write_op.cc:276] Timestamp: P: 1736650650765487 usec, L: 0
0112 02:57:38.822046 (+ 413us) write_op.cc:270] Start()
0112 02:57:38.822112 (+ 66us) write_op.cc:276] Timestamp: P: 1736650650795499 usec, L: 0
0112 02:57:38.822547 (+ 435us) write_op.cc:270] Start()
0112 02:57:38.822611 (+ 64us) write_op.cc:276] Timestamp: P: 1736650650822822 usec, L: 0
0112 02:57:38.823029 (+ 418us) write_op.cc:270] Start()
0112 02:57:38.823156 (+ 127us) write_op.cc:276] Timestamp: P: 1736650650873012 usec, L: 0
0112 02:57:38.823564 (+ 408us) write_op.cc:270] Start()
0112 02:57:38.823638 (+ 74us) write_op.cc:276] Timestamp: P: 1736650650934596 usec, L: 0
0112 02:57:38.824061 (+ 423us) write_op.cc:270] Start()
0112 02:57:38.824136 (+ 75us) write_op.cc:276] Timestamp: P: 1736650651080094 usec, L: 0
0112 02:57:38.824535 (+ 399us) write_op.cc:270] Start()
0112 02:57:38.824605 (+ 70us) write_op.cc:276] Timestamp: P: 1736650651455809 usec, L: 0
0112 02:57:38.825070 (+ 465us) write_op.cc:270] Start()
0112 02:57:38.825148 (+ 78us) write_op.cc:276] Timestamp: P: 1736650651463948 usec, L: 0
0112 02:57:38.825568 (+ 420us) write_op.cc:270] Start()
0112 02:57:38.825638 (+ 70us) write_op.cc:276] Timestamp: P: 1736650651468251 usec, L: 0
0112 02:57:38.826077 (+ 439us) write_op.cc:270] Start()
0112 02:57:38.826157 (+ 80us) write_op.cc:276] Timestamp: P: 1736650651629499 usec, L: 0
0112 02:57:38.875135 (+ 48978us) write_op.cc:270] Start()
0112 02:57:38.875272 (+ 137us) write_op.cc:276] Timestamp: P: 1736650651631953 usec, L: 0
0112 02:57:38.875928 (+ 656us) write_op.cc:270] Start()
0112 02:57:38.876028 (+ 100us) write_op.cc:276] Timestamp: P: 1736650651633815 usec, L: 0
0112 02:57:38.876531 (+ 503us) write_op.cc:270] Start()
0112 02:57:38.876614 (+ 83us) write_op.cc:276] Timestamp: P: 1736650651712273 usec, L: 0
0112 02:57:38.877151 (+ 537us) write_op.cc:270] Start()
0112 02:57:38.877235 (+ 84us) write_op.cc:276] Timestamp: P: 1736650651776247 usec, L: 0
0112 02:57:38.877712 (+ 477us) write_op.cc:270] Start()
0112 02:57:38.877793 (+ 81us) write_op.cc:276] Timestamp: P: 1736650651869484 usec, L: 0
0112 02:57:38.878248 (+ 455us) write_op.cc:270] Start()
0112 02:57:38.878339 (+ 91us) write_op.cc:276] Timestamp: P: 1736650651907730 usec, L: 0
0112 02:57:38.954345 (+ 76006us) write_op.cc:270] Start()
0112 02:57:38.954438 (+ 93us) write_op.cc:276] Timestamp: P: 1736650651938297 usec, L: 0
0112 02:57:38.954999 (+ 561us) write_op.cc:270] Start()
0112 02:57:38.955081 (+ 82us) write_op.cc:276] Timestamp: P: 1736650652180268 usec, L: 0
0112 02:57:38.962572 (+ 7491us) write_op.cc:270] Start()
0112 02:57:38.962666 (+ 94us) write_op.cc:276] Timestamp: P: 1736650652182690 usec, L: 0
0112 02:57:38.963213 (+ 547us) write_op.cc:270] Start()
0112 02:57:38.963286 (+ 73us) write_op.cc:276] Timestamp: P: 1736650652190884 usec, L: 0
0112 02:57:38.963696 (+ 410us) write_op.cc:270] Start()
0112 02:57:38.963766 (+ 70us) write_op.cc:276] Timestamp: P: 1736650652201770 usec, L: 0
0112 02:57:38.964337 (+ 571us) write_op.cc:270] Start()
0112 02:57:38.964415 (+ 78us) write_op.cc:276] Timestamp: P: 1736650652395013 usec, L: 0
0112 02:57:38.964895 (+ 480us) write_op.cc:270] Start()
0112 02:57:38.964974 (+ 79us) write_op.cc:276] Timestamp: P: 1736650652479667 usec, L: 0
0112 02:57:38.965422 (+ 448us) write_op.cc:270] Start()
0112 02:57:38.965506 (+ 84us) write_op.cc:276] Timestamp: P: 1736650652568626 usec, L: 0
0112 02:57:38.965992 (+ 486us) write_op.cc:270] Start()
0112 02:57:38.966058 (+ 66us) write_op.cc:276] Timestamp: P: 1736650652588581 usec, L: 0
0112 02:57:38.966482 (+ 424us) write_op.cc:270] Start()
0112 02:57:39.040300 (+ 73818us) write_op.cc:276] Timestamp: P: 1736650652668578 usec, L: 0
0112 02:57:39.041566 (+ 1266us) write_op.cc:270] Start()
0112 02:57:39.041767 (+ 201us) write_op.cc:276] Timestamp: P: 1736650652825597 usec, L: 0
0112 02:57:39.042771 (+ 1004us) write_op.cc:270] Start()
0112 02:57:39.042962 (+ 191us) write_op.cc:276] Timestamp: P: 1736650652851295 usec, L: 0
0112 02:57:39.043986 (+ 1024us) write_op.cc:270] Start()
0112 02:57:39.044190 (+ 204us) write_op.cc:276] Timestamp: P: 1736650652882283 usec, L: 0
0112 02:57:39.085067 (+ 40877us) write_op.cc:270] Start()
0112 02:57:39.086936 (+ 1869us) write_op.cc:276] Timestamp: P: 1736650652949194 usec, L: 0
0112 02:57:39.087581 (+ 645us) write_op.cc:270] Start()
0112 02:57:39.087659 (+ 78us) write_op.cc:276] Timestamp: P: 1736650653112500 usec, L: 0
0112 02:57:39.088128 (+ 469us) write_op.cc:270] Start()
0112 02:57:39.088197 (+ 69us) write_op.cc:276] Timestamp: P: 1736650653315461 usec, L: 0
0112 02:57:39.088607 (+ 410us) write_op.cc:270] Start()
0112 02:57:39.088691 (+ 84us) write_op.cc:276] Timestamp: P: 1736650654082047 usec, L: 0
0112 02:57:39.089133 (+ 442us) write_op.cc:270] Start()
0112 02:57:39.089235 (+ 102us) write_op.cc:276] Timestamp: P: 1736650654551422 usec, L: 0
0112 02:57:39.089616 (+ 381us) write_op.cc:270] Start()
0112 02:57:39.089678 (+ 62us) write_op.cc:276] Timestamp: P: 1736650654703824 usec, L: 0
0112 02:57:39.090067 (+ 389us) write_op.cc:270] Start()
0112 02:57:39.090150 (+ 83us) write_op.cc:276] Timestamp: P: 1736650654736927 usec, L: 0
0112 02:57:39.131150 (+ 41000us) write_op.cc:270] Start()
0112 02:57:39.131262 (+ 112us) write_op.cc:276] Timestamp: P: 1736650654739869 usec, L: 0
0112 02:57:39.131842 (+ 580us) write_op.cc:270] Start()
0112 02:57:39.131930 (+ 88us) write_op.cc:276] Timestamp: P: 1736650654775125 usec, L: 0
0112 02:57:39.132399 (+ 469us) write_op.cc:270] Start()
0112 02:57:39.132475 (+ 76us) write_op.cc:276] Timestamp: P: 1736650654856181 usec, L: 0
0112 02:57:39.132959 (+ 484us) write_op.cc:270] Start()
0112 02:57:39.133042 (+ 83us) write_op.cc:276] Timestamp: P: 1736650655274237 usec, L: 0
0112 02:57:39.156250 (+ 23208us) write_op.cc:270] Start()
0112 02:57:39.156407 (+ 157us) write_op.cc:276] Timestamp: P: 1736650655376887 usec, L: 0
0112 02:57:39.157461 (+ 1054us) write_op.cc:270] Start()
0112 02:57:39.157658 (+ 197us) write_op.cc:276] Timestamp: P: 1736650655432168 usec, L: 0
0112 02:57:39.158690 (+ 1032us) write_op.cc:270] Start()
0112 02:57:39.158841 (+ 151us) write_op.cc:276] Timestamp: P: 1736650655527984 usec, L: 0
0112 02:57:39.268202 (+109361us) write_op.cc:270] Start()
0112 02:57:39.268322 (+ 120us) write_op.cc:276] Timestamp: P: 1736650655573644 usec, L: 0
0112 02:57:39.274548 (+ 6226us) write_op.cc:270] Start()
0112 02:57:39.287541 (+ 12993us) write_op.cc:276] Timestamp: P: 1736650655603879 usec, L: 0
0112 02:57:39.288471 (+ 930us) write_op.cc:270] Start()
0112 02:57:39.288554 (+ 83us) write_op.cc:276] Timestamp: P: 1736650655621755 usec, L: 0
0112 02:57:39.289100 (+ 546us) write_op.cc:270] Start()
0112 02:57:39.289180 (+ 80us) write_op.cc:276] Timestamp: P: 1736650655701486 usec, L: 0
0112 02:57:39.289600 (+ 420us) write_op.cc:270] Start()
0112 02:57:39.289666 (+ 66us) write_op.cc:276] Timestamp: P: 1736650655802130 usec, L: 0
0112 02:57:39.319652 (+ 29986us) write_op.cc:270] Start()
0112 02:57:39.319739 (+ 87us) write_op.cc:276] Timestamp: P: 1736650655847244 usec, L: 0
0112 02:57:39.320211 (+ 472us) write_op.cc:270] Start()
0112 02:57:39.320285 (+ 74us) write_op.cc:276] Timestamp: P: 1736650656101381 usec, L: 0
0112 02:57:39.320750 (+ 465us) write_op.cc:270] Start()
0112 02:57:39.320822 (+ 72us) write_op.cc:276] Timestamp: P: 1736650656131007 usec, L: 0
0112 02:57:39.321267 (+ 445us) write_op.cc:270] Start()
0112 02:57:39.321342 (+ 75us) write_op.cc:276] Timestamp: P: 1736650656237230 usec, L: 0
0112 02:57:39.366795 (+ 45453us) write_op.cc:270] Start()
0112 02:57:39.366895 (+ 100us) write_op.cc:276] Timestamp: P: 1736650656283406 usec, L: 0
0112 02:57:39.371174 (+ 4279us) write_op.cc:270] Start()
0112 02:57:39.371471 (+ 297us) write_op.cc:276] Timestamp: P: 1736650656315444 usec, L: 0
0112 02:57:39.372641 (+ 1170us) write_op.cc:270] Start()
0112 02:57:39.372788 (+ 147us) write_op.cc:276] Timestamp: P: 1736650656409475 usec, L: 0
0112 02:57:39.373280 (+ 492us) write_op.cc:270] Start()
0112 02:57:39.373366 (+ 86us) write_op.cc:276] Timestamp: P: 1736650657441252 usec, L: 0
0112 02:57:39.522308 (+148942us) log.cc:844] Serialized 41839 byte log entry
0112 02:57:39.563091 (+ 40783us) raft_consensus.cc:1640] Marking committed up to 365
0112 02:57:39.679633 (+116542us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.679687 (+ 54us) write_op.cc:454] Released schema lock
0112 02:57:39.768739 (+ 89052us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.768764 (+ 25us) write_op.cc:454] Released schema lock
0112 02:57:39.770686 (+ 1922us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.770713 (+ 27us) write_op.cc:454] Released schema lock
0112 02:57:39.772682 (+ 1969us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.772707 (+ 25us) write_op.cc:454] Released schema lock
0112 02:57:39.776400 (+ 3693us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.776423 (+ 23us) write_op.cc:454] Released schema lock
0112 02:57:39.806634 (+ 30211us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.806666 (+ 32us) write_op.cc:454] Released schema lock
0112 02:57:39.808845 (+ 2179us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.808876 (+ 31us) write_op.cc:454] Released schema lock
0112 02:57:39.880360 (+ 71484us) raft_consensus.cc:1692] Filling consensus response to leader.
0112 02:57:39.921390 (+ 41030us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400040454. stack: 00007fe157858853 00007fe157858750 00007fe1573cd584 00000000004fcd24 00000000004fccb9 00007fe15e758d50 00007fe15e7dbf2a 00007fe1605aa4bc 00007fe1605b28dc 00007fe1605b2888 00007fe1605b2808 00007fe1605b27c3 00007fe1605b1a6c 00007fe15ec13486 00007fe15ec13418 00007fe15e7f24ea
0112 02:57:39.929432 (+ 8042us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0112 02:57:39.936522 (+ 7090us) raft_consensus.cc:1680] finished
0112 02:57:39.936572 (+ 50us) raft_consensus.cc:1686] UpdateReplicas() finished
0112 02:57:39.936979 (+ 407us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:57:38.815300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.815721 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.815885 (+ 164us) write_op.cc:435] Acquired schema lock
0112 02:57:38.815901 (+ 16us) tablet.cc:659] Decoding operations
0112 02:57:38.816162 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.816229 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.816252 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:38.816393 (+ 141us) tablet.cc:698] Row locks acquired
0112 02:57:38.816408 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:39.563516 (+747108us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.583316 (+ 19800us) write_op.cc:301] APPLY: starting
0112 02:57:39.583369 (+ 53us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.583448 (+ 79us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.583476 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.583624 (+ 148us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.583751 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:39.583868 (+ 117us) write_op.cc:312] APPLY: finished
0112 02:57:39.584038 (+ 170us) log.cc:844] Serialized 23 byte log entry
0112 02:57:39.584226 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.584283 (+ 57us) write_op.cc:454] Released schema lock
0112 02:57:39.584310 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:38.830653 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.831039 (+ 386us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.831063 (+ 24us) write_op.cc:435] Acquired schema lock
0112 02:57:38.831078 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:38.831505 (+ 427us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.831557 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.831581 (+ 24us) tablet.cc:682] Acquiring locks for 3 operations
0112 02:57:38.831850 (+ 269us) tablet.cc:698] Row locks acquired
0112 02:57:38.831878 (+ 28us) write_op.cc:260] PREPARE: finished
0112 02:57:39.680320 (+848442us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.706953 (+ 26633us) write_op.cc:301] APPLY: starting
0112 02:57:39.707033 (+ 80us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.707257 (+ 224us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.707288 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.707818 (+ 530us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.707968 (+ 150us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0112 02:57:39.708106 (+ 138us) write_op.cc:312] APPLY: finished
0112 02:57:39.708400 (+ 294us) log.cc:844] Serialized 35 byte log entry
0112 02:57:39.708757 (+ 357us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.708860 (+ 103us) write_op.cc:454] Released schema lock
0112 02:57:39.708893 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:39.709106 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.709126 (+ 20us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:38.832078 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.832404 (+ 326us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.832423 (+ 19us) write_op.cc:435] Acquired schema lock
0112 02:57:38.832436 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:38.832661 (+ 225us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.832710 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.832731 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:38.832848 (+ 117us) tablet.cc:698] Row locks acquired
0112 02:57:38.832877 (+ 29us) write_op.cc:260] PREPARE: finished
0112 02:57:39.680789 (+847912us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.706947 (+ 26158us) write_op.cc:301] APPLY: starting
0112 02:57:39.707111 (+ 164us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.707257 (+ 146us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.707288 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.718309 (+ 11021us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.718435 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:39.727935 (+ 9500us) write_op.cc:312] APPLY: finished
0112 02:57:39.728110 (+ 175us) log.cc:844] Serialized 23 byte log entry
0112 02:57:39.728298 (+ 188us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.728346 (+ 48us) write_op.cc:454] Released schema lock
0112 02:57:39.728371 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:39.728551 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.728572 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:38.833077 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.833409 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.833432 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:38.833447 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:38.833692 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.833738 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.833767 (+ 29us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.833940 (+ 173us) tablet.cc:698] Row locks acquired
0112 02:57:38.833955 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:39.700764 (+866809us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.709342 (+ 8578us) write_op.cc:301] APPLY: starting
0112 02:57:39.709385 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.709471 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.709487 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.709893 (+ 406us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.710002 (+ 109us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:39.710092 (+ 90us) write_op.cc:312] APPLY: finished
0112 02:57:39.710221 (+ 129us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.710310 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.710361 (+ 51us) write_op.cc:454] Released schema lock
0112 02:57:39.710384 (+ 23us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:39.766227 (+ 55843us) spinlock_profiling.cc:243] Waited 52.2 ms on lock 0x7b1800091300. stack: 00007fe157858853 00007fe157858750 00007fe1573cd584 00000000004fcd24 00000000004fccb9 00007fe16049d931 00007fe1604aa831 00007fe15e5f908c 00007fe15786d4ec 000000000058ec29 00007fe15788a35b 00007fe15788875f 00007fe15788a8c2 00007fe15ec3bd20 00007fe15ec3c2b9 00007fe15ec05e2a
0112 02:57:39.772420 (+ 6193us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.772441 (+ 21us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:38.834153 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.904984 (+ 70831us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.905033 (+ 49us) write_op.cc:435] Acquired schema lock
0112 02:57:38.905055 (+ 22us) tablet.cc:659] Decoding operations
0112 02:57:38.905434 (+ 379us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.905487 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.905515 (+ 28us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.905801 (+ 286us) tablet.cc:698] Row locks acquired
0112 02:57:38.905839 (+ 38us) write_op.cc:260] PREPARE: finished
0112 02:57:39.766526 (+860687us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.766794 (+ 268us) write_op.cc:301] APPLY: starting
0112 02:57:39.766854 (+ 60us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.767015 (+ 161us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.767036 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.767409 (+ 373us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.767659 (+ 250us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 02:57:39.767828 (+ 169us) write_op.cc:312] APPLY: finished
0112 02:57:39.771229 (+ 3401us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.771408 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.771508 (+ 100us) write_op.cc:454] Released schema lock
0112 02:57:39.771535 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:39.771712 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.771745 (+ 33us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:38.956561 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.956984 (+ 423us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.957007 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:57:38.957022 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:38.957342 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.957400 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.957424 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.957634 (+ 210us) tablet.cc:698] Row locks acquired
0112 02:57:38.957646 (+ 12us) write_op.cc:260] PREPARE: finished
0112 02:57:39.766840 (+809194us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.767091 (+ 251us) write_op.cc:301] APPLY: starting
0112 02:57:39.767132 (+ 41us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.767250 (+ 118us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.767275 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.767725 (+ 450us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.767900 (+ 175us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:39.767992 (+ 92us) write_op.cc:312] APPLY: finished
0112 02:57:39.768189 (+ 197us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.768328 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.768442 (+ 114us) write_op.cc:454] Released schema lock
0112 02:57:39.768472 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:38.957891 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.958248 (+ 357us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.958268 (+ 20us) write_op.cc:435] Acquired schema lock
0112 02:57:38.958281 (+ 13us) tablet.cc:659] Decoding operations
0112 02:57:38.958550 (+ 269us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.958596 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.958620 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.958805 (+ 185us) tablet.cc:698] Row locks acquired
0112 02:57:38.958823 (+ 18us) write_op.cc:260] PREPARE: finished
0112 02:57:39.769039 (+810216us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.769232 (+ 193us) write_op.cc:301] APPLY: starting
0112 02:57:39.769300 (+ 68us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.769420 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.769437 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.769832 (+ 395us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.769945 (+ 113us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:39.770059 (+ 114us) write_op.cc:312] APPLY: finished
0112 02:57:39.770201 (+ 142us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.770304 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.770428 (+ 124us) write_op.cc:454] Released schema lock
0112 02:57:39.770454 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:38.959025 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.959363 (+ 338us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.959384 (+ 21us) write_op.cc:435] Acquired schema lock
0112 02:57:38.959398 (+ 14us) tablet.cc:659] Decoding operations
0112 02:57:38.959639 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.959688 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.959716 (+ 28us) tablet.cc:682] Acquiring locks for 1 operations
0112 02:57:38.959846 (+ 130us) tablet.cc:698] Row locks acquired
0112 02:57:38.959857 (+ 11us) write_op.cc:260] PREPARE: finished
0112 02:57:39.770941 (+811084us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.771128 (+ 187us) write_op.cc:301] APPLY: starting
0112 02:57:39.771201 (+ 73us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.771298 (+ 97us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.771315 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.771432 (+ 117us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.771548 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:39.771662 (+ 114us) write_op.cc:312] APPLY: finished
0112 02:57:39.771818 (+ 156us) log.cc:844] Serialized 23 byte log entry
0112 02:57:39.771975 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.772203 (+ 228us) write_op.cc:454] Released schema lock
0112 02:57:39.772228 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:38.960084 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.960439 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.960476 (+ 37us) write_op.cc:435] Acquired schema lock
0112 02:57:38.960488 (+ 12us) tablet.cc:659] Decoding operations
0112 02:57:38.960742 (+ 254us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.960789 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.960818 (+ 29us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.961025 (+ 207us) tablet.cc:698] Row locks acquired
0112 02:57:38.961040 (+ 15us) write_op.cc:260] PREPARE: finished
0112 02:57:39.772922 (+811882us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.773352 (+ 430us) write_op.cc:301] APPLY: starting
0112 02:57:39.773396 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.773490 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.773508 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.773870 (+ 362us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.773987 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:57:39.774094 (+ 107us) write_op.cc:312] APPLY: finished
0112 02:57:39.774234 (+ 140us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.774440 (+ 206us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.774538 (+ 98us) write_op.cc:454] Released schema lock
0112 02:57:39.774575 (+ 37us) write_op.cc:341] FINISH: Updating metrics
0112 02:57:39.774773 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.774793 (+ 20us) write_op.cc:454] Released schema lock
Related trace 'op':
0112 02:57:38.961289 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:38.961650 (+ 361us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:57:38.961675 (+ 25us) write_op.cc:435] Acquired schema lock
0112 02:57:38.961690 (+ 15us) tablet.cc:659] Decoding operations
0112 02:57:38.962013 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:57:38.962066 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0112 02:57:38.962096 (+ 30us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:57:38.962301 (+ 205us) tablet.cc:698] Row locks acquired
0112 02:57:38.962319 (+ 18us) write_op.cc:260] PREPARE: finished
0112 02:57:39.773175 (+810856us) op_driver.cc:464] REPLICATION: finished
0112 02:57:39.774992 (+ 1817us) write_op.cc:301] APPLY: starting
0112 02:57:39.775031 (+ 39us) tablet.cc:1364] starting BulkCheckPresence
0112 02:57:39.775144 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0112 02:57:39.775162 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:57:39.775506 (+ 344us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:57:39.775671 (+ 165us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0112 02:57:39.775777 (+ 106us) write_op.cc:312] APPLY: finished
0112 02:57:39.775905 (+ 128us) log.cc:844] Serialized 29 byte log entry
0112 02:57:39.776082 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:57:39.776141 (+ 59us) write_op.cc:454] Released schema lock
0112 02:57:39.776170 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0112 02:57:39.034256 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:57:39.035206 (+
I20250112 02:57:41.876206 22144 tablet_bootstrap.cc:654] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.923362 22145 tablet_bootstrap.cc:492] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:41.923966 23334 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Loaded TSK: 0
I20250112 02:57:41.925099 23334 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250112 02:57:41.928696 21378 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4716677cfad74c0dafe18c55796aeae5 consensus state has no leader
I20250112 02:57:41.929325 22145 tablet_bootstrap.cc:654] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:41.938179 21377 catalog_manager.cc:4872] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): tablet 922b22f0b1284bc4a732371a61cb1ed4 (table default.Table_7d88c7e3065640629e9d17b64174c9e4 [id=f1b12e4291bd474983056b14b9a070f6]) successfully deleted
I20250112 02:57:34.634214 22883 rpcz_store.cc:276] 0112 02:57:30.696210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:30.696402 (+ 192us) service_pool.cc:225] Handling call
0112 02:57:31.606922 (+910520us) raft_consensus.cc:1466] Updating replica for 292 ops
0112 02:57:31.639632 (+ 32710us) raft_consensus.cc:1529] Early marking committed up to index 0
0112 02:57:31.639688 (+ 56us) raft_consensus.cc:1534] Triggering prepare for 292 ops
0112 02:57:31.640442 (+ 754us) write_op.cc:270] Start()
0112 02:57:31.640587 (+ 145us) write_op.cc:276] Timestamp: P: 1736650632655457 usec, L: 0
0112 02:57:31.773903 (+133316us) write_op.cc:270] Start()
0112 02:57:31.773994 (+ 91us) write_op.cc:276] Timestamp: P: 1736650632749201 usec, L: 0
0112 02:57:31.774591 (+ 597us) write_op.cc:270] Start()
0112 02:57:31.776668 (+ 2077us) write_op.cc:276] Timestamp: P: 1736650632788705 usec, L: 0
0112 02:57:31.777170 (+ 502us) write_op.cc:270] Start()
0112 02:57:31.777246 (+ 76us) write_op.cc:276] Timestamp: P: 1736650632827570 usec, L: 0
0112 02:57:31.777753 (+ 507us) write_op.cc:270] Start()
0112 02:57:31.777836 (+ 83us) write_op.cc:276] Timestamp: P: 1736650632840790 usec, L: 0
0112 02:57:31.778314 (+ 478us) write_op.cc:270] Start()
0112 02:57:31.778402 (+ 88us) write_op.cc:276] Timestamp: P: 1736650632867997 usec, L: 0
0112 02:57:31.779007 (+ 605us) write_op.cc:270] Start()
0112 02:57:31.779151 (+ 144us) write_op.cc:276] Timestamp: P: 1736650632870366 usec, L: 0
0112 02:57:31.779686 (+ 535us) write_op.cc:270] Start()
0112 02:57:31.779773 (+ 87us) write_op.cc:276] Timestamp: P: 1736650632895218 usec, L: 0
0112 02:57:31.780231 (+ 458us) write_op.cc:270] Start()
0112 02:57:31.780314 (+ 83us) write_op.cc:276] Timestamp: P: 1736650632898019 usec, L: 0
0112 02:57:31.780855 (+ 541us) write_op.cc:270] Start()
0112 02:57:31.780946 (+ 91us) write_op.cc:276] Timestamp: P: 1736650632948187 usec, L: 0
0112 02:57:31.784565 (+ 3619us) write_op.cc:270] Start()
0112 02:57:31.784671 (+ 106us) write_op.cc:276] Timestamp: P: 1736650632990876 usec, L: 0
0112 02:57:31.785171 (+ 500us) write_op.cc:270] Start()
0112 02:57:31.785272 (+ 101us) write_op.cc:276] Timestamp: P: 1736650633031560 usec, L: 0
0112 02:57:31.785727 (+ 455us) write_op.cc:270] Start()
0112 02:57:31.785798 (+ 71us) write_op.cc:276] Timestamp: P: 1736650633093689 usec, L: 0
0112 02:57:31.786222 (+ 424us) write_op.cc:270] Start()
0112 02:57:31.786295 (+ 73us) write_op.cc:276] Timestamp: P: 1736650633173382 usec, L: 0
0112 02:57:31.786743 (+ 448us) write_op.cc:270] Start()
0112 02:57:31.786817 (+ 74us) write_op.cc:276] Timestamp: P: 1736650633219164 usec, L: 0
0112 02:57:31.787246 (+ 429us) write_op.cc:270] Start()
0112 02:57:31.787327 (+ 81us) write_op.cc:276] Timestamp: P: 1736650633250826 usec, L: 0
0112 02:57:31.787825 (+ 498us) write_op.cc:270] Start()
0112 02:57:31.787900 (+ 75us) write_op.cc:276] Timestamp: P: 1736650633261098 usec, L: 0
0112 02:57:31.788353 (+ 453us) write_op.cc:270] Start()
0112 02:57:31.788429 (+ 76us) write_op.cc:276] Timestamp: P: 1736650633308466 usec, L: 0
0112 02:57:31.788890 (+ 461us) write_op.cc:270] Start()
0112 02:57:31.788968 (+ 78us) write_op.cc:276] Timestamp: P: 1736650633320674 usec, L: 0
0112 02:57:31.801261 (+ 12293us) write_op.cc:270] Start()
0112 02:57:31.801382 (+ 121us) write_op.cc:276] Timestamp: P: 1736650633322950 usec, L: 0
0112 02:57:31.801896 (+ 514us) write_op.cc:270] Start()
0112 02:57:31.801977 (+ 81us) write_op.cc:276] Timestamp: P: 1736650633325598 usec, L: 0
0112 02:57:31.802416 (+ 439us) write_op.cc:270] Start()
0112 02:57:31.802541 (+ 125us) write_op.cc:276] Timestamp: P: 1736650633419321 usec, L: 0
0112 02:57:31.803092 (+ 551us) write_op.cc:270] Start()
0112 02:57:31.803178 (+ 86us) write_op.cc:276] Timestamp: P: 1736650633462375 usec, L: 0
0112 02:57:31.803665 (+ 487us) write_op.cc:270] Start()
0112 02:57:31.803801 (+ 136us) write_op.cc:276] Timestamp: P: 1736650633465645 usec, L: 0
0112 02:57:31.804477 (+ 676us) write_op.cc:270] Start()
0112 02:57:31.804567 (+ 90us) write_op.cc:276] Timestamp: P: 1736650633522423 usec, L: 0
0112 02:57:31.805031 (+ 464us) write_op.cc:270] Start()
0112 02:57:31.805117 (+ 86us) write_op.cc:276] Timestamp: P: 1736650633535160 usec, L: 0
0112 02:57:31.805532 (+ 415us) write_op.cc:270] Start()
0112 02:57:31.805609 (+ 77us) write_op.cc:276] Timestamp: P: 1736650633569679 usec, L: 0
0112 02:57:31.806025 (+ 416us) write_op.cc:270] Start()
0112 02:57:31.806097 (+ 72us) write_op.cc:276] Timestamp: P: 1736650633577822 usec, L: 0
0112 02:57:31.806568 (+ 471us) write_op.cc:270] Start()
0112 02:57:31.806662 (+ 94us) write_op.cc:276] Timestamp: P: 1736650633648700 usec, L: 0
0112 02:57:31.807161 (+ 499us) write_op.cc:270] Start()
0112 02:57:31.807250 (+ 89us) write_op.cc:276] Timestamp: P: 1736650633659152 usec, L: 0
0112 02:57:31.807749 (+ 499us) write_op.cc:270] Start()
0112 02:57:31.807842 (+ 93us) write_op.cc:276] Timestamp: P: 1736650633690368 usec, L: 0
0112 02:57:31.808316 (+ 474us) write_op.cc:270] Start()
0112 02:57:31.808529 (+ 213us) write_op.cc:276] Timestamp: P: 1736650633700950 usec, L: 0
0112 02:57:31.809063 (+ 534us) write_op.cc:270] Start()
0112 02:57:31.809134 (+ 71us) write_op.cc:276] Timestamp: P: 1736650633716754 usec, L: 0
0112 02:57:31.809555 (+ 421us) write_op.cc:270] Start()
0112 02:57:31.809628 (+ 73us) write_op.cc:276] Timestamp: P: 1736650633781128 usec, L: 0
0112 02:57:31.810105 (+ 477us) write_op.cc:270] Start()
0112 02:57:31.810224 (+ 119us) write_op.cc:276] Timestamp: P: 1736650633795381 usec, L: 0
0112 02:57:31.810746 (+ 522us) write_op.cc:270] Start()
0112 02:57:31.810835 (+ 89us) write_op.cc:276] Timestamp: P: 1736650633832761 usec, L: 0
0112 02:57:31.811305 (+ 470us) write_op.cc:270] Start()
0112 02:57:31.811397 (+ 92us) write_op.cc:276] Timestamp: P: 1736650633892932 usec, L: 0
0112 02:57:31.816976 (+ 5579us) write_op.cc:270] Start()
0112 02:57:31.817045 (+ 69us) write_op.cc:276] Timestamp: P: 1736650633924437 usec, L: 0
0112 02:57:31.817513 (+ 468us) write_op.cc:270] Start()
0112 02:57:31.817603 (+ 90us) write_op.cc:276] Timestamp: P: 1736650633969262 usec, L: 0
0112 02:57:31.818048 (+ 445us) write_op.cc:270] Start()
0112 02:57:31.818155 (+ 107us) write_op.cc:276] Timestamp: P: 1736650633977533 usec, L: 0
0112 02:57:31.818663 (+ 508us) write_op.cc:270] Start()
0112 02:57:31.818724 (+ 61us) write_op.cc:276] Timestamp: P: 1736650634046205 usec, L: 0
0112 02:57:31.819103 (+ 379us) write_op.cc:270] Start()
0112 02:57:31.819166 (+ 63us) write_op.cc:276] Timestamp: P: 1736650634090454 usec, L: 0
0112 02:57:31.819563 (+ 397us) write_op.cc:270] Start()
0112 02:57:31.819632 (+ 69us) write_op.cc:276] Timestamp: P: 1736650634105735 usec, L: 0
0112 02:57:31.820009 (+ 377us) write_op.cc:270] Start()
0112 02:57:31.820071 (+ 62us) write_op.cc:276] Timestamp: P: 1736650634114572 usec, L: 0
0112 02:57:31.820464 (+ 393us) write_op.cc:270] Start()
0112 02:57:31.820529 (+ 65us) write_op.cc:276] Timestamp: P: 1736650634197961 usec, L: 0
0112 02:57:31.820904 (+ 375us) write_op.cc:270] Start()
0112 02:57:31.820970 (+ 66us) write_op.cc:276] Timestamp: P: 1736650634217155 usec, L: 0
0112 02:57:31.821361 (+ 391us) write_op.cc:270] Start()
0112 02:57:31.821443 (+ 82us) write_op.cc:276] Timestamp: P: 1736650634227691 usec, L: 0
0112 02:57:31.851067 (+ 29624us) write_op.cc:270] Start()
0112 02:57:31.851217 (+ 150us) write_op.cc:276] Timestamp: P: 1736650634233594 usec, L: 0
0112 02:57:31.851813 (+ 596us) write_op.cc:270] Start()
0112 02:57:31.851890 (+ 77us) write_op.cc:276] Timestamp: P: 1736650634236388 usec, L: 0
0112 02:57:31.852307 (+ 417us) write_op.cc:270] Start()
0112 02:57:31.856669 (+ 4362us) write_op.cc:276] Timestamp: P: 1736650634306218 usec, L: 0
0112 02:57:31.857184 (+ 515us) write_op.cc:270] Start()
0112 02:57:31.857262 (+ 78us) write_op.cc:276] Timestamp: P: 1736650634395027 usec, L: 0
0112 02:57:31.857719 (+ 457us) write_op.cc:270] Start()
0112 02:57:31.857800 (+ 81us) write_op.cc:276] Timestamp: P: 1736650634396992 usec, L: 0
0112 02:57:31.858373 (+ 573us) write_op.cc:270] Start()
0112 02:57:31.858454 (+ 81us) write_op.cc:276] Timestamp: P: 1736650634465493 usec, L: 0
0112 02:57:31.858964 (+ 510us) write_op.cc:270] Start()
0112 02:57:31.859041 (+ 77us) write_op.cc:276] Timestamp: P: 1736650634490039 usec, L: 0
0112 02:57:31.859495 (+ 454us) write_op.cc:270] Start()
0112 02:57:31.859587 (+ 92us) write_op.cc:276] Timestamp: P: 1736650634499843 usec, L: 0
0112 02:57:31.860074 (+ 487us) write_op.cc:270] Start()
0112 02:57:31.860152 (+ 78us) write_op.cc:276] Timestamp: P: 1736650634563544 usec, L: 0
0112 02:57:31.860609 (+ 457us) write_op.cc:270] Start()
0112 02:57:31.860687 (+ 78us) write_op.cc:276] Timestamp: P: 1736650634585579 usec, L: 0
0112 02:57:31.861152 (+ 465us) write_op.cc:270] Start()
0112 02:57:31.861230 (+ 78us) write_op.cc:276] Timestamp: P: 1736650634619766 usec, L: 0
0112 02:57:31.861687 (+ 457us) write_op.cc:270] Start()
0112 02:57:31.861777 (+ 90us) write_op.cc:276] Timestamp: P: 1736650634677408 usec, L: 0
0112 02:57:31.862235 (+ 458us) write_op.cc:270] Start()
0112 02:57:31.862414 (+ 179us) write_op.cc:276] Timestamp: P: 1736650634739452 usec, L: 0
0112 02:57:31.960164 (+ 97750us) write_op.cc:270] Start()
0112 02:57:31.960270 (+ 106us) write_op.cc:276] Timestamp: P: 1736650634808891 usec, L: 0
0112 02:57:31.960805 (+ 535us) write_op.cc:270] Start()
0112 02:57:31.960889 (+ 84us) write_op.cc:276] Timestamp: P: 1736650634868675 usec, L: 0
0112 02:57:31.961348 (+ 459us) write_op.cc:270] Start()
0112 02:57:31.961424 (+ 76us) write_op.cc:276] Timestamp: P: 1736650634870443 usec, L: 0
0112 02:57:31.961891 (+ 467us) write_op.cc:270] Start()
0112 02:57:31.961972 (+ 81us) write_op.cc:276] Timestamp: P: 1736650634912296 usec, L: 0
0112 02:57:31.971812 (+ 9840us) write_op.cc:270] Start()
0112 02:57:31.971906 (+ 94us) write_op.cc:276] Timestamp: P: 1736650634985994 usec, L: 0
0112 02:57:31.972448 (+ 542us) write_op.cc:270] Start()
0112 02:57:31.972524 (+ 76us) write_op.cc:276] Timestamp: P: 1736650635013611 usec, L: 0
0112 02:57:31.972945 (+ 421us) write_op.cc:270] Start()
0112 02:57:31.973013 (+ 68us) write_op.cc:276] Timestamp: P: 1736650635023973 usec, L: 0
0112 02:57:31.973441 (+ 428us) write_op.cc:270] Start()
0112 02:57:31.973511 (+ 70us) write_op.cc:276] Timestamp: P: 1736650635060238 usec, L: 0
0112 02:57:31.973920 (+ 409us) write_op.cc:270] Start()
0112 02:57:31.973991 (+ 71us) write_op.cc:276] Timestamp: P: 1736650635115004 usec, L: 0
0112 02:57:31.974438 (+ 447us) write_op.cc:270] Start()
0112 02:57:31.976128 (+ 1690us) write_op.cc:276] Timestamp: P: 1736650635209664 usec, L: 0
0112 02:57:31.976612 (+ 484us) write_op.cc:270] Start()
0112 02:57:31.976687 (+ 75us) write_op.cc:276] Timestamp: P: 1736650635228627 usec, L: 0
0112 02:57:31.977132 (+ 445us) write_op.cc:270] Start()
0112 02:57:31.977219 (+ 87us) write_op.cc:276] Timestamp: P: 1736650635267552 usec, L: 0
0112 02:57:31.977671 (+ 452us) write_op.cc:270] Start()
0112 02:57:31.977745 (+ 74us) write_op.cc:276] Timestamp: P: 1736650635328532 usec, L: 0
0112 02:57:31.978225 (+ 480us) write_op.cc:270] Start()
0112 02:57:31.978303 (+ 78us) write_op.cc:276] Timestamp: P: 1736650635448134 usec, L: 0
0112 02:57:31.981064 (+ 2761us) write_op.cc:270] Start()
0112 02:57:31.981183 (+ 119us) write_op.cc:276] Timestamp: P: 1736650635548187 usec, L: 0
0112 02:57:31.987330 (+ 6147us) write_op.cc:270] Start()
0112 02:57:31.987447 (+ 117us) write_op.cc:276] Timestamp: P: 1736650635577520 usec, L: 0
0112 02:57:31.987959 (+ 512us) write_op.cc:270] Start()
0112 02:57:31.988041 (+ 82us) write_op.cc:276] Timestamp: P: 1736650635602986 usec, L: 0
0112 02:57:31.988530 (+ 489us) write_op.cc:270] Start()
0112 02:57:31.988621 (+ 91us) write_op.cc:276] Timestamp: P: 1736650635636662 usec, L: 0
0112 02:57:31.995519 (+ 6898us) write_op.cc:270] Start()
0112 02:57:31.995610 (+ 91us) write_op.cc:276] Timestamp: P: 1736650635684104 usec, L: 0
0112 02:57:31.996099 (+ 489us) write_op.cc:270] Start()
0112 02:57:31.996184 (+ 85us) write_op.cc:276] Timestamp: P: 1736650635862819 usec, L: 0
0112 02:57:31.996628 (+ 444us) write_op.cc:270] Start()
0112 02:57:31.996704 (+ 76us) write_op.cc:276] Timestamp: P: 1736650635918558 usec, L: 0
0112 02:57:31.997157 (+ 453us) write_op.cc:270] Start()
0112 02:57:31.997234 (+ 77us) write_op.cc:276] Timestamp: P: 1736650635982929 usec, L: 0
0112 02:57:31.997708 (+ 474us) write_op.cc:270] Start()
0112 02:57:31.997788 (+ 80us) write_op.cc:276] Timestamp: P: 1736650636033285 usec, L: 0
0112 02:57:31.998244 (+ 456us) write_op.cc:270] Start()
0112 02:57:31.998323 (+ 79us) write_op.cc:276] Timestamp: P: 1736650636069382 usec, L: 0
0112 02:57:32.002902 (+ 4579us) write_op.cc:270] Start()
0112 02:57:32.002992 (+ 90us) write_op.cc:276] Timestamp: P: 1736650636071439 usec, L: 0
0112 02:57:32.003484 (+ 492us) write_op.cc:270] Start()
0112 02:57:32.003558 (+ 74us) write_op.cc:276] Timestamp: P: 1736650636093809 usec, L: 0
0112 02:57:32.004050 (+ 492us) write_op.cc:270] Start()
0112 02:57:32.004131 (+ 81us) write_op.cc:276] Timestamp: P: 1736650636125838 usec, L: 0
0112 02:57:32.004609 (+ 478us) write_op.cc:270] Start()
0112 02:57:32.004713 (+ 104us) write_op.cc:276] Timestamp: P: 1736650636176876 usec, L: 0
0112 02:57:32.005255 (+ 542us) write_op.cc:270] Start()
0112 02:57:32.005343 (+ 88us) write_op.cc:276] Timestamp: P: 1736650636265478 usec, L: 0
0112 02:57:32.005845 (+ 502us) write_op.cc:270] Start()
0112 02:57:32.005932 (+ 87us) write_op.cc:276] Timestamp: P: 1736650636373135 usec, L: 0
0112 02:57:32.006424 (+ 492us) write_op.cc:270] Start()
0112 02:57:32.037687 (+ 31263us) write_op.cc:276] Timestamp: P: 1736650636694319 usec, L: 0
0112 02:57:32.038385 (+ 698us) write_op.cc:270] Start()
0112 02:57:32.038490 (+ 105us) write_op.cc:276] Timestamp: P: 1736650636742296 usec, L: 0
0112 02:57:32.039004 (+ 514us) write_op.cc:270] Start()
0112 02:57:32.039084 (+ 80us) write_op.cc:276] Timestamp: P: 1736650636864577 usec, L: 0
0112 02:57:32.039576 (+ 492us) write_op.cc:270] Start()
0112 02:57:32.039656 (+ 80us) write_op.cc:276] Timestamp: P: 1736650636867779 usec, L: 0
0112 02:57:32.040106 (+ 450us) write_op.cc:270] Start()
0112 02:57:32.040195 (+ 89us) write_op.cc:276] Timestamp: P: 1736650636926545 usec, L: 0
0112 02:57:32.040741 (+ 546us) write_op.cc:270] Start()
0112 02:57:32.040822 (+ 81us) write_op.cc:276] Timestamp: P: 1736650636955482 usec, L: 0
0112 02:57:32.041268 (+ 446us) write_op.cc:270] Start()
0112 02:57:32.041342 (+ 74us) write_op.cc:276] Timestamp: P: 1736650636999645 usec, L: 0
0112 02:57:32.041798 (+ 456us) write_op.cc:270] Start()
0112 02:57:32.041990 (+ 192us) write_op.cc:276] Timestamp: P: 1736650637031981 usec, L: 0
0112 02:57:32.062977 (+ 20987us) write_op.cc:270] Start()
0112 02:57:32.063090 (+ 113us) write_op.cc:276] Timestamp: P: 1736650637155748 usec, L: 0
0112 02:57:32.063689 (+ 599us) write_op.cc:270] Start()
0112 02:57:32.063768 (+ 79us) write_op.cc:276] Timestamp: P: 1736650637197453 usec, L: 0
0112 02:57:32.066337 (+ 2569us) write_op.cc:270] Start()
0112 02:57:32.066420 (+ 83us) write_op.cc:276] Timestamp: P: 1736650637250875 usec, L: 0
0112 02:57:32.067258 (+ 838us) write_op.cc:270] Start()
0112 02:57:32.067334 (+ 76us) write_op.cc:276] Timestamp: P: 1736650637309805 usec, L: 0
0112 02:57:32.067848 (+ 514us) write_op.cc:270] Start()
0112 02:57:32.067923 (+ 75us) write_op.cc:276] Timestamp: P: 1736650637376605 usec, L: 0
0112 02:57:32.068377 (+ 454us) write_op.cc:270] Start()
0112 02:57:32.068454 (+ 77us) write_op.cc:276] Timestamp: P: 1736650637399558 usec, L: 0
0112 02:57:32.081203 (+ 12749us) write_op.cc:270] Start()
0112 02:57:32.081290 (+ 87us) write_op.cc:276] Timestamp: P: 1736650637472486 usec, L: 0
0112 02:57:32.083376 (+ 2086us) write_op.cc:270] Start()
0112 02:57:32.083460 (+ 84us) write_op.cc:276] Timestamp: P: 1736650637474368 usec, L: 0
0112 02:57:32.083919 (+ 459us) write_op.cc:270] Start()
0112 02:57:32.083995 (+ 76us) write_op.cc:276] Timestamp: P: 1736650637483788 usec, L: 0
0112 02:57:32.084455 (+ 460us) write_op.cc:270] Start()
0112 02:57:32.084544 (+ 89us) write_op.cc:276] Timestamp: P: 1736650637586424 usec, L: 0
0112 02:57:32.084975 (+ 431us) write_op.cc:270] Start()
0112 02:57:32.085049 (+ 74us) write_op.cc:276] Timestamp: P: 1736650637612741 usec, L: 0
0112 02:57:32.085522 (+ 473us) write_op.cc:270] Start()
0112 02:57:32.085599 (+ 77us) write_op.cc:276] Timestamp: P: 1736650637697644 usec, L: 0
0112 02:57:32.086047 (+ 448us) write_op.cc:270] Start()
0112 02:57:32.086136 (+ 89us) write_op.cc:276] Timestamp: P: 1736650637774123 usec, L: 0
0112 02:57:32.094744 (+ 8608us) write_op.cc:270] Start()
0112 02:57:32.094824 (+ 80us) write_op.cc:276] Timestamp: P: 1736650637810525 usec, L: 0
0112 02:57:32.099091 (+ 4267us) write_op.cc:270] Start()
0112 02:57:32.099178 (+ 87us) write_op.cc:276] Timestamp: P: 1736650637823178 usec, L: 0
0112 02:57:32.099593 (+ 415us) write_op.cc:270] Start()
0112 02:57:32.099673 (+ 80us) write_op.cc:276] Timestamp: P: 1736650637861847 usec, L: 0
0112 02:57:32.100072 (+ 399us) write_op.cc:270] Start()
0112 02:57:32.100136 (+ 64us) write_op.cc:276] Timestamp: P: 1736650637900495 usec, L: 0
0112 02:57:32.100528 (+ 392us) write_op.cc:270] Start()
0112 02:57:32.100604 (+ 76us) write_op.cc:276] Timestamp: P: 1736650637938475 usec, L: 0
0112 02:57:32.100973 (+ 369us) write_op.cc:270] Start()
0112 02:57:32.101048 (+ 75us) write_op.cc:276] Timestamp: P: 1736650637979179 usec, L: 0
0112 02:57:32.101454 (+ 406us) write_op.cc:270] Start()
0112 02:57:32.101518 (+ 64us) write_op.cc:276] Timestamp: P: 1736650638189433 usec, L: 0
0112 02:57:32.101895 (+ 377us) write_op.cc:270] Start()
0112 02:57:32.109904 (+ 8009us) write_op.cc:276] Timestamp: P: 1736650638205920 usec, L: 0
0112 02:57:32.110408 (+ 504us) write_op.cc:270] Start()
0112 02:57:32.110483 (+ 75us) write_op.cc:276] Timestamp: P: 1736650638270909 usec, L: 0
0112 02:57:32.110917 (+ 434us) write_op.cc:270] Start()
0112 02:57:32.110990 (+ 73us) write_op.cc:276] Timestamp: P: 1736650638329665 usec, L: 0
0112 02:57:32.111471 (+ 481us) write_op.cc:270] Start()
0112 02:57:32.111542 (+ 71us) write_op.cc:276] Timestamp: P: 1736650638435326 usec, L: 0
0112 02:57:32.111932 (+ 390us) write_op.cc:270] Start()
0112 02:57:32.112010 (+ 78us) write_op.cc:276] Timestamp: P: 1736650638618451 usec, L: 0
0112 02:57:32.112454 (+ 444us) write_op.cc:270] Start()
0112 02:57:32.112519 (+ 65us) write_op.cc:276] Timestamp: P: 1736650638650317 usec, L: 0
0112 02:57:32.112905 (+ 386us) write_op.cc:270] Start()
0112 02:57:32.112970 (+ 65us) write_op.cc:276] Timestamp: P: 1736650638706779 usec, L: 0
0112 02:57:32.113457 (+ 487us) write_op.cc:270] Start()
0112 02:57:32.113520 (+ 63us) write_op.cc:276] Timestamp: P: 1736650638728354 usec, L: 0
0112 02:57:32.113915 (+ 395us) write_op.cc:270] Start()
0112 02:57:32.113993 (+ 78us) write_op.cc:276] Timestamp: P: 1736650638730119 usec, L: 0
0112 02:57:32.114462 (+ 469us) write_op.cc:270] Start()
0112 02:57:32.131777 (+ 17315us) write_op.cc:276] Timestamp: P: 1736650638888642 usec, L: 0
0112 02:57:32.132581 (+ 804us) write_op.cc:270] Start()
0112 02:57:32.132655 (+ 74us) write_op.cc:276] Timestamp: P: 1736650638926287 usec, L: 0
0112 02:57:32.133137 (+ 482us) write_op.cc:270] Start()
0112 02:57:32.133210 (+ 73us) write_op.cc:276] Timestamp: P: 1736650639075703 usec, L: 0
0112 02:57:32.133626 (+ 416us) write_op.cc:270] Start()
0112 02:57:32.133770 (+ 144us) write_op.cc:276] Timestamp: P: 1736650639117347 usec, L: 0
0112 02:57:32.134224 (+ 454us) write_op.cc:270] Start()
0112 02:57:32.134303 (+ 79us) write_op.cc:276] Timestamp: P: 1736650639156401 usec, L: 0
0112 02:57:32.136689 (+ 2386us) write_op.cc:270] Start()
0112 02:57:32.136755 (+ 66us) write_op.cc:276] Timestamp: P: 1736650639227086 usec, L: 0
0112 02:57:32.137188 (+ 433us) write_op.cc:270] Start()
0112 02:57:32.137275 (+ 87us) write_op.cc:276] Timestamp: P: 1736650639281334 usec, L: 0
0112 02:57:32.137788 (+ 513us) write_op.cc:270] Start()
0112 02:57:32.137856 (+ 68us) write_op.cc:276] Timestamp: P: 1736650639283183 usec, L: 0
0112 02:57:32.138267 (+ 411us) write_op.cc:270] Start()
0112 02:57:32.138334 (+ 67us) write_op.cc:276] Timestamp: P: 1736650639285021 usec, L: 0
0112 02:57:32.138791 (+ 457us) write_op.cc:270] Start()
0112 02:57:32.138865 (+ 74us) write_op.cc:276] Timestamp: P: 1736650639393809 usec, L: 0
0112 02:57:32.139331 (+ 466us) write_op.cc:270] Start()
0112 02:57:32.139403 (+ 72us) write_op.cc:276] Timestamp: P: 1736650639399075 usec, L: 0
0112 02:57:32.139830 (+ 427us) write_op.cc:270] Start()
0112 02:57:32.139913 (+ 83us) write_op.cc:276] Timestamp: P: 1736650639499122 usec, L: 0
0112 02:57:32.140357 (+ 444us) write_op.cc:270] Start()
0112 02:57:32.140430 (+ 73us) write_op.cc:276] Timestamp: P: 1736650639503784 usec, L: 0
0112 02:57:32.171639 (+ 31209us) write_op.cc:270] Start()
0112 02:57:32.171722 (+ 83us) write_op.cc:276] Timestamp: P: 1736650639514417 usec, L: 0
0112 02:57:32.172181 (+ 459us) write_op.cc:270] Start()
0112 02:57:32.172247 (+ 66us) write_op.cc:276] Timestamp: P: 1736650639581509 usec, L: 0
0112 02:57:32.172679 (+ 432us) write_op.cc:270] Start()
0112 02:57:32.172745 (+ 66us) write_op.cc:276] Timestamp: P: 1736650639583229 usec, L: 0
0112 02:57:32.173151 (+ 406us) write_op.cc:270] Start()
0112 02:57:32.173218 (+ 67us) write_op.cc:276] Timestamp: P: 1736650639593522 usec, L: 0
0112 02:57:32.173603 (+ 385us) write_op.cc:270] Start()
0112 02:57:32.173670 (+ 67us) write_op.cc:276] Timestamp: P: 1736650639635240 usec, L: 0
0112 02:57:32.174064 (+ 394us) write_op.cc:270] Start()
0112 02:57:32.174126 (+ 62us) write_op.cc:276] Timestamp: P: 1736650639665836 usec, L: 0
0112 02:57:32.181041 (+ 6915us) write_op.cc:270] Start()
0112 02:57:32.181127 (+ 86us) write_op.cc:276] Timestamp: P: 1736650639849240 usec, L: 0
0112 02:57:32.181592 (+ 465us) write_op.cc:270] Start()
0112 02:57:32.181676 (+ 84us) write_op.cc:276] Timestamp: P: 1736650639885018 usec, L: 0
0112 02:57:32.182904 (+ 1228us) write_op.cc:270] Start()
0112 02:57:32.182972 (+ 68us) write_op.cc:276] Timestamp: P: 1736650639910367 usec, L: 0
0112 02:57:32.183405 (+ 433us) write_op.cc:270] Start()
0112 02:57:32.183471 (+ 66us) write_op.cc:276] Timestamp: P: 1736650639939330 usec, L: 0
0112 02:57:32.183871 (+ 400us) write_op.cc:270] Start()
0112 02:57:32.183938 (+ 67us) write_op.cc:276] Timestamp: P: 1736650639978444 usec, L: 0
0112 02:57:32.184334 (+ 396us) write_op.cc:270] Start()
0112 02:57:32.184399 (+ 65us) write_op.cc:276] Timestamp: P: 1736650640025775 usec, L: 0
0112 02:57:32.184764 (+ 365us) write_op.cc:270] Start()
0112 02:57:32.184833 (+ 69us) write_op.cc:276] Timestamp: P: 1736650640050351 usec, L: 0
0112 02:57:32.185255 (+ 422us) write_op.cc:270] Start()
0112 02:57:32.185329 (+ 74us) write_op.cc:276] Timestamp: P: 1736650640371464 usec, L: 0
0112 02:57:32.185724 (+ 395us) write_op.cc:270] Start()
0112 02:57:32.185797 (+ 73us) write_op.cc:276] Timestamp: P: 1736650640511845 usec, L: 0
0112 02:57:32.186391 (+ 594us) write_op.cc:270] Start()
0112 02:57:32.186458 (+ 67us) write_op.cc:276] Timestamp: P: 1736650640542767 usec, L: 0
0112 02:57:32.284168 (+ 97710us) write_op.cc:270] Start()
0112 02:57:32.284252 (+ 84us) write_op.cc:276] Timestamp: P: 1736650640547017 usec, L: 0
0112 02:57:32.284719 (+ 467us) write_op.cc:270] Start()
0112 02:57:32.284784 (+ 65us) write_op.cc:276] Timestamp: P: 1736650640577281 usec, L: 0
0112 02:57:32.285344 (+ 560us) write_op.cc:270] Start()
0112 02:57:32.285409 (+ 65us) write_op.cc:276] Timestamp: P: 1736650640582361 usec, L: 0
0112 02:57:32.285802 (+ 393us) write_op.cc:270] Start()
0112 02:57:32.285867 (+ 65us) write_op.cc:276] Timestamp: P: 1736650640584638 usec, L: 0
0112 02:57:32.286245 (+ 378us) write_op.cc:270] Start()
0112 02:57:32.286307 (+ 62us) write_op.cc:276] Timestamp: P: 1736650640646507 usec, L: 0
0112 02:57:32.286729 (+ 422us) write_op.cc:270] Start()
0112 02:57:32.286803 (+ 74us) write_op.cc:276] Timestamp: P: 1736650640836953 usec, L: 0
0112 02:57:32.287220 (+ 417us) write_op.cc:270] Start()
0112 02:57:32.287310 (+ 90us) write_op.cc:276] Timestamp: P: 1736650640893221 usec, L: 0
0112 02:57:32.287726 (+ 416us) write_op.cc:270] Start()
0112 02:57:32.287802 (+ 76us) write_op.cc:276] Timestamp: P: 1736650640922599 usec, L: 0
0112 02:57:32.288226 (+ 424us) write_op.cc:270] Start()
0112 02:57:32.288296 (+ 70us) write_op.cc:276] Timestamp: P: 1736650641050271 usec, L: 0
0112 02:57:32.301245 (+ 12949us) write_op.cc:270] Start()
0112 02:57:32.301340 (+ 95us) write_op.cc:276] Timestamp: P: 1736650641053736 usec, L: 0
0112 02:57:32.306699 (+ 5359us) write_op.cc:270] Start()
0112 02:57:32.306781 (+ 82us) write_op.cc:276] Timestamp: P: 1736650641062098 usec, L: 0
0112 02:57:32.307234 (+ 453us) write_op.cc:270] Start()
0112 02:57:32.321003 (+ 13769us) write_op.cc:276] Timestamp: P: 1736650641202341 usec, L: 0
0112 02:57:32.321539 (+ 536us) write_op.cc:270] Start()
0112 02:57:32.321617 (+ 78us) write_op.cc:276] Timestamp: P: 1736650641258433 usec, L: 0
0112 02:57:32.322077 (+ 460us) write_op.cc:270] Start()
0112 02:57:32.322146 (+ 69us) write_op.cc:276] Timestamp: P: 1736650641260149 usec, L: 0
0112 02:57:32.322559 (+ 413us) write_op.cc:270] Start()
0112 02:57:32.322630 (+ 71us) write_op.cc:276] Timestamp: P: 1736650641261957 usec, L: 0
0112 02:57:32.323015 (+ 385us) write_op.cc:270] Start()
0112 02:57:32.323077 (+ 62us) write_op.cc:276] Timestamp: P: 1736650641418676 usec, L: 0
0112 02:57:32.352457 (+ 29380us) write_op.cc:270] Start()
0112 02:57:32.352580 (+ 123us) write_op.cc:276] Timestamp: P: 1736650641547680 usec, L: 0
0112 02:57:32.353211 (+ 631us) write_op.cc:270] Start()
0112 02:57:32.353309 (+ 98us) write_op.cc:276] Timestamp: P: 1736650641812975 usec, L: 0
0112 02:57:32.353777 (+ 468us) write_op.cc:270] Start()
0112 02:57:32.353859 (+ 82us) write_op.cc:276] Timestamp: P: 1736650641858805 usec, L: 0
0112 02:57:32.354442 (+ 583us) write_op.cc:270] Start()
0112 02:57:32.354556 (+ 114us) write_op.cc:276] Timestamp: P: 1736650641868951 usec, L: 0
0112 02:57:32.355051 (+ 495us) write_op.cc:270] Start()
0112 02:57:32.355138 (+ 87us) write_op.cc:276] Timestamp: P: 1736650641962506 usec, L: 0
0112 02:57:32.355572 (+ 434us) write_op.cc:270] Start()
0112 02:57:32.355635 (+ 63us) write_op.cc:276] Timestamp: P: 1736650642001988 usec, L: 0
0112 02:57:32.355991 (+ 356us) write_op.cc:270] Start()
0112 02:57:32.356051 (+ 60us) write_op.cc:276] Timestamp: P: 1736650642005000 usec, L: 0
0112 02:57:32.356408 (+ 357us) write_op.cc:270] Start()
0112 02:57:32.356470 (+ 62us) write_op.cc:276] Timestamp: P: 1736650642071659 usec, L: 0
0112 02:57:32.356995 (+ 525us) write_op.cc:270] Start()
0112 02:57:32.357080 (+ 85us) write_op.cc:276] Timestamp: P: 1736650642151559 usec, L: 0
0112 02:57:32.357558 (+ 478us) write_op.cc:270] Start()
0112 02:57:32.357632 (+ 74us) write_op.cc:276] Timestamp: P: 1736650642159395 usec, L: 0
0112 02:57:32.380503 (+ 22871us) write_op.cc:270] Start()
0112 02:57:32.380589 (+ 86us) write_op.cc:276] Timestamp: P: 1736650642252942 usec, L: 0
0112 02:57:32.381029 (+ 440us) write_op.cc:270] Start()
0112 02:57:32.381095 (+ 66us) write_op.cc:276] Timestamp: P: 1736650642913126 usec, L: 0
0112 02:57:32.393462 (+ 12367us) write_op.cc:270] Start()
0112 02:57:32.393543 (+ 81us) write_op.cc:276] Timestamp: P: 1736650642957562 usec, L: 0
0112 02:57:32.394213 (+ 670us) write_op.cc:270] Start()
0112 02:57:32.394298 (+ 85us) write_op.cc:276] Timestamp: P: 1736650643061663 usec, L: 0
0112 02:57:32.394722 (+ 424us) write_op.cc:270] Start()
0112 02:57:32.394788 (+ 66us) write_op.cc:276] Timestamp: P: 1736650643146544 usec, L: 0
0112 02:57:32.395203 (+ 415us) write_op.cc:270] Start()
0112 02:57:32.395268 (+ 65us) write_op.cc:276] Timestamp: P: 1736650643180962 usec, L: 0
0112 02:57:32.397548 (+ 2280us) write_op.cc:270] Start()
0112 02:57:32.397616 (+ 68us) write_op.cc:276] Timestamp: P: 1736650643223553 usec, L: 0
0112 02:57:32.398036 (+ 420us) write_op.cc:270] Start()
0112 02:57:32.398111 (+ 75us) write_op.cc:276] Timestamp: P: 1736650643247221 usec, L: 0
0112 02:57:32.398535 (+ 424us) write_op.cc:270] Start()
0112 02:57:32.398594 (+ 59us) write_op.cc:276] Timestamp: P: 1736650643289938 usec, L: 0
0112 02:57:32.398944 (+ 350us) write_op.cc:270] Start()
0112 02:57:32.398999 (+ 55us) write_op.cc:276] Timestamp: P: 1736650643315502 usec, L: 0
0112 02:57:32.399375 (+ 376us) write_op.cc:270] Start()
0112 02:57:32.399441 (+ 66us) write_op.cc:276] Timestamp: P: 1736650643348547 usec, L: 0
0112 02:57:32.399831 (+ 390us) write_op.cc:270] Start()
0112 02:57:32.399907 (+ 76us) write_op.cc:276] Timestamp: P: 1736650643430004 usec, L: 0
0112 02:57:32.424611 (+ 24704us) write_op.cc:270] Start()
0112 02:57:32.424696 (+ 85us) write_op.cc:276] Timestamp: P: 1736650643507181 usec, L: 0
0112 02:57:32.425171 (+ 475us) write_op.cc:270] Start()
0112 02:57:32.425239 (+ 68us) write_op.cc:276] Timestamp: P: 1736650643537247 usec, L: 0
0112 02:57:32.425664 (+ 425us) write_op.cc:270] Start()
0112 02:57:32.425735 (+ 71us) write_op.cc:276] Timestamp: P: 17366506
W20250112 02:57:41.945873 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 2601daad8ba742ec88ccfe603440e8e7 because the tablet was not found. No further retry: Not found: Tablet not found: 2601daad8ba742ec88ccfe603440e8e7
W20250112 02:57:41.946432 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 5bd0d70fb0fe4a179b1ea6fa23d180c9 because the tablet was not found. No further retry: Not found: Tablet not found: 5bd0d70fb0fe4a179b1ea6fa23d180c9
W20250112 02:57:41.946985 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 247f0ef0ada44d9d957c8ff42b0da4b5 because the tablet was not found. No further retry: Not found: Tablet not found: 247f0ef0ada44d9d957c8ff42b0da4b5
W20250112 02:57:41.947469 21377 catalog_manager.cc:4839] TS e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): delete failed for tablet 443aa544f6df4d50a78c9effd306e216 because the tablet was not found. No further retry: Not found: Tablet not found: 443aa544f6df4d50a78c9effd306e216
I20250112 02:57:41.949632 22144 tablet_bootstrap.cc:492] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:41.950040 22144 ts_tablet_manager.cc:1397] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.119s user 0.006s sys 0.003s
I20250112 02:57:41.952190 22144 raft_consensus.cc:357] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.952834 22144 raft_consensus.cc:383] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.953091 22144 raft_consensus.cc:738] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca160b7cffb4ae287ea077e7352ffc9, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.953795 22144 consensus_queue.cc:260] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.958247 23334 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:57:41.968034 23482 raft_consensus.cc:491] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:41.968402 23482 raft_consensus.cc:513] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.970005 23482 leader_election.cc:290] T 58c49391d2ed4c3fa84fa2d071b06094 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:41.972018 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58c49391d2ed4c3fa84fa2d071b06094" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:41.981070 22145 tablet_bootstrap.cc:492] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:41.981492 22145 ts_tablet_manager.cc:1397] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.004s
I20250112 02:57:41.983672 22145 raft_consensus.cc:357] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.984277 22145 raft_consensus.cc:383] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:41.984526 22145 raft_consensus.cc:738] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:41.985067 22145 consensus_queue.cc:260] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:41.989737 23475 ts_tablet_manager.cc:1918] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:57:42.003033 22145 ts_tablet_manager.cc:1428] T 56c7f07da89c4994a57c23ebdd8061bf P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20250112 02:57:42.003142 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd]
I20250112 02:57:42.004344 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740]
I20250112 02:57:42.007422 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae]
I20250112 02:57:42.008559 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e]
I20250112 02:57:42.009589 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb]
I20250112 02:57:42.010632 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0]
I20250112 02:57:42.011648 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043]
I20250112 02:57:42.013957 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748]
I20250112 02:57:42.015075 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2]
I20250112 02:57:42.016881 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf]
I20250112 02:57:42.017894 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952]
I20250112 02:57:42.019003 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041]
I20250112 02:57:42.020656 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab]
I20250112 02:57:42.022902 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac]
I20250112 02:57:42.023069 22145 tablet_bootstrap.cc:492] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:42.024421 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963]
I20250112 02:57:42.025708 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4]
I20250112 02:57:42.026878 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152]
I20250112 02:57:42.027909 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d]
I20250112 02:57:42.028299 22145 tablet_bootstrap.cc:654] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:42.029006 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7]
I20250112 02:57:42.030128 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e]
I20250112 02:57:42.031270 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_03e04d34d58e463e93d16e858752d3cb [id=30ad19fdfc1246ebbeaf73fff91f925e]
I20250112 02:57:42.033114 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59]
I20250112 02:57:42.035693 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618]
I20250112 02:57:42.036808 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c]
I20250112 02:57:42.038617 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c]
I20250112 02:57:42.040431 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db]
I20250112 02:57:42.040525 22145 tablet_bootstrap.cc:492] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:42.040838 22145 ts_tablet_manager.cc:1397] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.001s
I20250112 02:57:42.041486 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce]
I20250112 02:57:42.042869 22145 raft_consensus.cc:357] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.043460 22145 raft_consensus.cc:383] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:42.043725 22145 raft_consensus.cc:738] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:42.044265 22145 consensus_queue.cc:260] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.045769 22145 ts_tablet_manager.cc:1428] T da54d9dbab2f416785ec0379095bcf3e P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 02:57:42.046571 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5]
I20250112 02:57:42.048256 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96]
I20250112 02:57:42.049325 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597]
I20250112 02:57:42.051005 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d]
I20250112 02:57:42.027653 22144 ts_tablet_manager.cc:1428] T a345af15e7664d01a184193ddb51572b P cca160b7cffb4ae287ea077e7352ffc9: Time spent starting tablet: real 0.077s user 0.002s sys 0.003s
I20250112 02:57:42.052259 22144 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9: Bootstrap starting.
I20250112 02:57:42.065989 22144 tablet_bootstrap.cc:654] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:42.070771 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f]
I20250112 02:57:42.071925 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde]
I20250112 02:57:42.075654 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a]
I20250112 02:57:42.076747 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3]
I20250112 02:57:42.079126 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd]
I20250112 02:57:42.080308 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3]
I20250112 02:57:42.082041 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b]
I20250112 02:57:42.083115 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d]
I20250112 02:57:42.084242 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680]
I20250112 02:57:42.085314 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b]
I20250112 02:57:42.071147 23475 log.cc:1198] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/f445a4bf43db4571b6d3a4afe8fe823a
I20250112 02:57:42.086292 23475 ts_tablet_manager.cc:1939] T f445a4bf43db4571b6d3a4afe8fe823a P cca160b7cffb4ae287ea077e7352ffc9: Deleting consensus metadata
I20250112 02:57:42.090611 23475 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35272 (request call id 277) took 1035 ms. Trace:
I20250112 02:57:42.092972 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0]
I20250112 02:57:42.094805 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a]
I20250112 02:57:42.095988 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d]
I20250112 02:57:42.106582 23475 rpcz_store.cc:276] 0112 02:57:41.055130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:57:41.055245 (+ 115us) service_pool.cc:225] Handling call
0112 02:57:41.830671 (+775426us) ts_tablet_manager.cc:1145] Deleting tablet f445a4bf43db4571b6d3a4afe8fe823a
0112 02:57:41.842544 (+ 11873us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0112 02:57:41.927786 (+ 85242us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:41.989514 (+ 61728us) tablet_metadata.cc:641] Metadata flushed
0112 02:57:42.090501 (+100987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":774936}
I20250112 02:57:42.107543 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7]
I20250112 02:57:42.107853 23475 tablet_replica.cc:331] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: stopping tablet replica
I20250112 02:57:42.107932 21378 catalog_manager.cc:4872] TS cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): tablet f445a4bf43db4571b6d3a4afe8fe823a (table default.Table_77fe18e65c724b129a710891e2e7e123 [id=7a50bffc5ac74966b8b07fd0cd5211b9]) successfully deleted
I20250112 02:57:42.108640 23475 raft_consensus.cc:2238] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:57:42.108847 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_181a7bdabe584055b44baf7ff5ab440f [id=8dbf245ab5354db7a93b54749e978c65]
I20250112 02:57:42.109079 23475 raft_consensus.cc:2267] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:57:42.109576 23482 raft_consensus.cc:491] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:42.110072 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669]
I20250112 02:57:42.109987 23482 raft_consensus.cc:513] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.111289 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f]
I20250112 02:57:42.112423 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9]
I20250112 02:57:42.113576 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064]
I20250112 02:57:42.115402 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2]
I20250112 02:57:42.117329 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4]
I20250112 02:57:42.117133 22041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334ed386f72d4fca870ade9d6a44589a" candidate_uuid: "cca160b7cffb4ae287ea077e7352ffc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" is_pre_election: true
I20250112 02:57:42.118346 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2]
I20250112 02:57:42.119403 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158]
I20250112 02:57:42.111858 23482 leader_election.cc:290] T 334ed386f72d4fca870ade9d6a44589a P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
I20250112 02:57:42.120455 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9]
I20250112 02:57:42.121503 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61]
I20250112 02:57:42.122629 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268]
I20250112 02:57:42.123792 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5]
I20250112 02:57:42.124913 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2]
I20250112 02:57:42.127630 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8]
I20250112 02:57:42.127830 22145 tablet_bootstrap.cc:492] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:42.128767 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43]
I20250112 02:57:42.130352 23474 raft_consensus.cc:491] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:42.131064 23474 raft_consensus.cc:513] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.134009 23474 leader_election.cc:290] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:42.134179 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6c28220edf4598af19ba21e3e7b7d3" candidate_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cca160b7cffb4ae287ea077e7352ffc9" is_pre_election: true
I20250112 02:57:42.130141 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822]
I20250112 02:57:42.135406 23475 ts_tablet_manager.cc:1905] T 4e2b248224f9416faf4bd83a853ec918 P cca160b7cffb4ae287ea077e7352ffc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:57:42.135574 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f]
W20250112 02:57:42.135674 21980 leader_election.cc:343] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:57:42.136705 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18]
I20250112 02:57:42.134375 22145 tablet_bootstrap.cc:654] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:42.138751 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c]
I20250112 02:57:42.139982 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d]
I20250112 02:57:42.141808 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0]
I20250112 02:57:42.143638 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682]
I20250112 02:57:42.145462 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec]
I20250112 02:57:42.146553 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f]
I20250112 02:57:42.147648 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575]
I20250112 02:57:42.148787 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143]
I20250112 02:57:42.149881 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86]
I20250112 02:57:42.151661 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4]
I20250112 02:57:42.152845 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815]
I20250112 02:57:42.152974 23482 raft_consensus.cc:491] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:57:42.153388 21376 connection.cc:537] client connection to 127.20.217.129:42085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:42.153415 23482 raft_consensus.cc:513] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.154994 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0]
I20250112 02:57:42.155501 23482 leader_election.cc:290] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085), e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307)
W20250112 02:57:42.156155 21837 connection.cc:537] server connection from 127.20.217.129:43957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:42.157377 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718]
W20250112 02:57:42.158572 21377 connection.cc:537] server connection from 127.20.217.129:43111 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:57:42.159515 21835 connection.cc:700] client connection to 127.20.217.129:42085 send error: Network error: sendmsg error: Connection reset by peer (error 104)
W20250112 02:57:42.159905 21835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: sendmsg error: Connection reset by peer (error 104)
W20250112 02:57:42.160018 21836 leader_election.cc:343] T 922b22f0b1284bc4a732371a61cb1ed4 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0406c08e9714e87bdd7fd6b66c9756d (127.20.217.131:39307): Not found: Tablet not found: 922b22f0b1284bc4a732371a61cb1ed4
I20250112 02:57:42.160507 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd]
I20250112 02:57:42.161947 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779]
W20250112 02:57:42.162786 21533 connection.cc:537] server connection from 127.20.217.129:52427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:57:42.163458 21978 connection.cc:537] client connection to 127.20.217.129:42085 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:57:42.163892 21978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:42.164186 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481]
I20250112 02:57:42.165585 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5]
I20250112 02:57:42.166770 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e]
I20250112 02:57:42.167153 22144 tablet_bootstrap.cc:492] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9: No bootstrap required, opened a new log
I20250112 02:57:42.167557 22144 ts_tablet_manager.cc:1397] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9: Time spent bootstrapping tablet: real 0.115s user 0.005s sys 0.005s
I20250112 02:57:42.167891 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169]
I20250112 02:57:42.168982 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac]
I20250112 02:57:42.169662 22144 raft_consensus.cc:357] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.170222 22144 raft_consensus.cc:383] T 7c9b3e6c84484575acd46e61dd1ea28a P cca160b7cffb4ae287ea077e7352ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:42.174989 22145 tablet_bootstrap.cc:492] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:42.175385 22145 ts_tablet_manager.cc:1397] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.005s
I20250112 02:57:42.177815 22145 raft_consensus.cc:357] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
W20250112 02:57:42.178193 21835 consensus_peers.cc:487] T 557de3de9a4d4d0bac1dc16f66ab6b01 P cca160b7cffb4ae287ea077e7352ffc9 -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:42.178480 22145 raft_consensus.cc:383] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250112 02:57:42.178609 21835 consensus_peers.cc:487] T 247f0ef0ada44d9d957c8ff42b0da4b5 P cca160b7cffb4ae287ea077e7352ffc9 -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:42.178771 22145 raft_consensus.cc:738] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:42.178929 21835 leader_election.cc:336] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.179304 21835 leader_election.cc:304] T 13edf2c56e8e4254b75e0eb85fb88427 P cca160b7cffb4ae287ea077e7352ffc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cca160b7cffb4ae287ea077e7352ffc9; no voters: 1c2770701ca54288956df9d910d7d3d2, e0406c08e9714e87bdd7fd6b66c9756d
I20250112 02:57:42.179594 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21808
I20250112 02:57:42.179410 22145 consensus_queue.cc:260] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.182960 22145 ts_tablet_manager.cc:1428] T d1e0548276474b68919cb10fbcfcdf04 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250112 02:57:42.183830 22145 tablet_bootstrap.cc:492] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:42.184123 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb]
I20250112 02:57:42.185388 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a]
I20250112 02:57:42.188640 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c]
I20250112 02:57:42.189885 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48]
I20250112 02:57:42.190950 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330]
I20250112 02:57:42.194108 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633]
W20250112 02:57:42.199169 21376 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.129:42085, user_credentials={real_user=slave}} blocked reactor thread for 45446.6us
I20250112 02:57:42.202483 22145 tablet_bootstrap.cc:654] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
I20250112 02:57:42.209030 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d]
I20250112 02:57:42.210038 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f]
I20250112 02:57:42.211093 23334 catalog_manager.cc:670] Loaded metadata for table default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12]
W20250112 02:57:42.212169 21376 catalog_manager.cc:4615] TS 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84f7700b8cce4261a32f2009d1481b99: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
W20250112 02:57:42.217805 21978 leader_election.cc:336] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.218209 21978 leader_election.cc:304] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.218976 23474 raft_consensus.cc:2743] T eab717cb5d794d15b9b51881b86501fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.219221 21978 leader_election.cc:336] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.219597 21978 leader_election.cc:304] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.220175 23474 raft_consensus.cc:2743] T 03d2021384bb41b58a66fcb7afaeaca8 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.220356 21978 leader_election.cc:336] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.220712 21978 leader_election.cc:304] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.221315 23474 raft_consensus.cc:2743] T 62253be16a1a4b2cb7d50264e82478fc P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.221536 21978 consensus_peers.cc:487] T 65eb8c99917247edbdbf5de42f3b505c P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.221925 21978 leader_election.cc:336] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.222210 21978 leader_election.cc:304] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.222754 23474 raft_consensus.cc:2743] T 2f6c28220edf4598af19ba21e3e7b7d3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.226122 21978 consensus_peers.cc:487] T 1c20f2a3dfbc4de2bde9491e17d6ba9a P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.226610 21978 consensus_peers.cc:487] T a345af15e7664d01a184193ddb51572b P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.227023 21978 consensus_peers.cc:487] T a90643c97b5e424a8bf3c262e3e5429e P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.227406 21978 consensus_peers.cc:487] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.227800 21978 consensus_peers.cc:487] T f47e14720a704a19ac3aea8723e13c28 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.228142 21978 leader_election.cc:336] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.228473 21978 leader_election.cc:304] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.229095 23474 raft_consensus.cc:2743] T 750246f0a0c549229d4e7a87cd6ddfd9 P e0406c08e9714e87bdd7fd6b66c9756d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.229828 21978 leader_election.cc:336] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.230209 21978 leader_election.cc:304] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.230861 23474 raft_consensus.cc:2743] T 779badaa22e6404aaaf1818e86228bd4 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.233036 21978 leader_election.cc:336] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.235900 22145 tablet_bootstrap.cc:492] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:42.236256 22145 ts_tablet_manager.cc:1397] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.010s
I20250112 02:57:42.233502 21978 leader_election.cc:304] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.246693 23474 raft_consensus.cc:2743] T a08141e3615244a9b266cd197d9b9ef3 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.246472 21978 leader_election.cc:336] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.247565 21978 leader_election.cc:304] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.248214 23474 raft_consensus.cc:2743] T 1bbce2d946cd47398a3d0a1f72ca1678 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.249054 21978 leader_election.cc:336] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.249423 21978 leader_election.cc:304] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.250092 23474 raft_consensus.cc:2743] T b32e6564d4dd4d1b9b25a14266b3dc04 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.251400 21978 consensus_peers.cc:487] T 118ef6d7c16941598177b92a94c4b472 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.251920 21978 consensus_peers.cc:487] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.252405 21978 leader_election.cc:336] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.252769 21978 leader_election.cc:304] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.252807 22145 raft_consensus.cc:357] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.253396 23474 raft_consensus.cc:2743] T 5c3fd2d14dbf4bc98e42a04e33c66348 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:57:42.253484 22145 raft_consensus.cc:383] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:42.253773 22145 raft_consensus.cc:738] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
W20250112 02:57:42.254658 21978 consensus_peers.cc:487] T 6e922b68d33b4a76a260816475ee6a3c P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.255090 21978 leader_election.cc:336] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.255457 21978 leader_election.cc:304] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.256139 23474 raft_consensus.cc:2743] T 08e25cf536254f3ea9d2c1e97a32b85b P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.258304 21978 consensus_peers.cc:487] T c80b7813b50d4739a76776d5d16434f3 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Couldn't send request to peer 1c2770701ca54288956df9d910d7d3d2. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:42.269140 23474 raft_consensus.cc:491] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:57:42.269618 23474 raft_consensus.cc:513] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.271620 23474 leader_election.cc:290] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673), 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085)
I20250112 02:57:42.254400 22145 consensus_queue.cc:260] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.275913 22145 ts_tablet_manager.cc:1428] T 7634a1d8de9b495599b83f15eaa91c89 P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.039s user 0.003s sys 0.003s
I20250112 02:57:42.276691 22145 tablet_bootstrap.cc:492] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:42.291894 22145 tablet_bootstrap.cc:654] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:42.298310 21978 leader_election.cc:336] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 1c2770701ca54288956df9d910d7d3d2 (127.20.217.129:42085): Network error: Client connection negotiation failed: client connection to 127.20.217.129:42085: connect: Connection refused (error 111)
I20250112 02:57:42.323717 22145 tablet_bootstrap.cc:492] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d: No bootstrap required, opened a new log
I20250112 02:57:42.324124 22145 ts_tablet_manager.cc:1397] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.001s
I20250112 02:57:42.333890 22145 raft_consensus.cc:357] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.334580 22145 raft_consensus.cc:383] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:57:42.334821 22145 raft_consensus.cc:738] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0406c08e9714e87bdd7fd6b66c9756d, State: Initialized, Role: FOLLOWER
I20250112 02:57:42.335383 22145 consensus_queue.cc:260] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cca160b7cffb4ae287ea077e7352ffc9" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 36673 } } peers { permanent_uuid: "1c2770701ca54288956df9d910d7d3d2" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 42085 } } peers { permanent_uuid: "e0406c08e9714e87bdd7fd6b66c9756d" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 39307 } }
I20250112 02:57:42.337847 22145 ts_tablet_manager.cc:1428] T af0c74dd3303445b9e9aef1f2d7af62a P e0406c08e9714e87bdd7fd6b66c9756d: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20250112 02:57:42.351274 23334 tablet_loader.cc:96] loaded metadata for tablet 030d48c4421e45f0900705c4b6a4f4ed (table default.Table_661154f6ad374f1f809b070c6274bd4d [id=3a3c494ea37e44a38eff597eac9bcd0c])
I20250112 02:57:42.353164 23334 tablet_loader.cc:96] loaded metadata for tablet 05ba915163894cc6aa0cfb144dd49d97 (table default.Table_d0fc06e00a274dc99d145cadb6530068 [id=7ecb59e52f944ce281cecb9525fba9fd])
I20250112 02:57:42.355484 23334 tablet_loader.cc:96] loaded metadata for tablet 08e25cf536254f3ea9d2c1e97a32b85b (table default.Table_02aed393b2e34f1db7553def2ae78759 [id=808dc550e95b46149fadc679b780e58b])
I20250112 02:57:42.364562 23334 tablet_loader.cc:96] loaded metadata for tablet 0df734ba87264abd803237a5bbb32da2 (table default.Table_3cc13a5a45e94ddf8fc3406f7c3dbc00 [id=ea31ec23822144d891b4c999b4d49169])
I20250112 02:57:42.365629 23334 tablet_loader.cc:96] loaded metadata for tablet 118ef6d7c16941598177b92a94c4b472 (table default.Table_aa7c0fcb28e54a549bbc2c16662666a3 [id=acb82bd3e7bf4f8f9b58e5c684dbec43])
I20250112 02:57:42.367923 23334 tablet_loader.cc:96] loaded metadata for tablet 13edf2c56e8e4254b75e0eb85fb88427 (table default.Table_e5f44414856346b8a9b001fe2d81f67f [id=23926f918b1746b8a080adef8c13b963])
I20250112 02:57:42.369767 23334 tablet_loader.cc:96] loaded metadata for tablet 1800d0eb7a9a48d295aca16607291add (table default.Table_4dcdc6c673294f60a1b60f8517bded49 [id=96df1b8c0560443981f1df06a75cc9c9])
I20250112 02:57:42.370961 23334 tablet_loader.cc:96] loaded metadata for tablet 1bbce2d946cd47398a3d0a1f72ca1678 (table default.Table_eab8f1f56fa94fb6be7fbd91911184bb [id=a1537985ccdf4326b98c3c920c42d158])
I20250112 02:57:42.372162 23334 tablet_loader.cc:96] loaded metadata for tablet 1c20f2a3dfbc4de2bde9491e17d6ba9a (table default.Table_948d7e0db47d4a47b2d6e6ec21baf292 [id=2fa952f6cb4e4fd5a40b77000c30d75e])
I20250112 02:57:42.375280 22145 tablet_bootstrap.cc:492] T 67236f9e5d11467486e81393dd01e0df P e0406c08e9714e87bdd7fd6b66c9756d: Bootstrap starting.
I20250112 02:57:42.391171 23334 tablet_loader.cc:96] loaded metadata for tablet 1cacc670998c4452bd3c03d99a62ad82 (table default.Table_426ae97685174521adbbaa140dad7cdc [id=aa5f944d36354a8ab55604e43d441268])
I20250112 02:57:42.392122 23334 tablet_loader.cc:96] loaded metadata for tablet 1db9ff39ff2341199549155a2db0aaba (table default.Table_8a6b1335ab9d4d8697519f270c8ac7ac [id=12de7f5fcf8943b98d4d80753d15b043])
W20250112 02:57:42.394040 21980 connection.cc:537] client connection to 127.20.217.130:36673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:57:42.399017 23501 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:57:42.395610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.217.130:36673 (local address 127.20.217.131:35529)
0112 02:57:42.396951 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0112 02:57:42.397010 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0112 02:57:42.397400 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 02:57:42.398174 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.217.130:36673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":825}
I20250112 02:57:42.400166 22145 tablet_bootstrap.cc:654] T 67236f9e5d11467486e81393dd01e0df P e0406c08e9714e87bdd7fd6b66c9756d: Neither blocks nor log segments found. Creating new log.
W20250112 02:57:42.402688 21980 consensus_peers.cc:487] T 21dc0755b38746a39bd773780e48cda5 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.130:36673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250112 02:57:42.403060 21980 leader_election.cc:336] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Network error: Client connection negotiation failed: client connection to 127.20.217.130:36673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:57:42.403391 21980 leader_election.cc:304] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0406c08e9714e87bdd7fd6b66c9756d; no voters: 1c2770701ca54288956df9d910d7d3d2, cca160b7cffb4ae287ea077e7352ffc9
I20250112 02:57:42.404008 23474 raft_consensus.cc:2743] T d94cda34ff2e44cfa543362a007f1400 P e0406c08e9714e87bdd7fd6b66c9756d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:57:42.404142 21980 consensus_peers.cc:487] T 43c413e1f8fe48039c4dd1b066fdd857 P e0406c08e9714e87bdd7fd6b66c9756d -> Peer cca160b7cffb4ae287ea077e7352ffc9 (127.20.217.130:36673): Couldn't send request to peer cca160b7cffb4ae287ea077e7352ffc9. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.130:36673: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:57:42.406397 23334 tablet_loader.cc:96] loaded metadata for tablet 1f21e3c769364b58bd371aac338db1f2 (table default.Table_817027c4f6374ecfb969e968afe83e5e [id=95345c3b8b944685ac3e34018cfbe38f])
I20250112 02:57:42.410920 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21957
I20250112 02:57:42.414348 23334 tablet_loader.cc:96] loaded metadata for tablet 21dc0755b38746a39bd773780e48cda5 (table default.Table_8f4aa29e292a4231bda19d00c4db1718 [id=c52787432d37482d9ed338ed0ed9e682])
I20250112 02:57:42.415776 23334 tablet_loader.cc:96] loaded metadata for tablet 22aac4332d0e476d9758004972ef8ab2 (table default.Table_91b38432112c4a8392b137502ce37881 [id=f49bfb986e05441cb2acd6dc84479330])
I20250112 02:57:42.422544 23334 tablet_loader.cc:96] loaded metadata for tablet 2722e4001ba44a7998f4ba7682bd3c44 (table default.Table_24f9f706cc244d319aa60e1f687d5325 [id=01455b7b2dbf4f3884c113e0a85b9ccd])
I20250112 02:57:42.425544 23334 tablet_loader.cc:96] loaded metadata for tablet 2cbf5e2a26474c98b9cb48b5bbdceafb (table default.Table_99128d2e95814f319baf8037cf642ca5 [id=f26eccc887824b91b9aded594fce2b9c])
I20250112 02:57:42.426914 23334 tablet_loader.cc:96] loaded metadata for tablet 2d793344526a404e8a737e7fb628e521 (table default.Table_d19f37f2c90c4162a8fb2e731a73d7f7 [id=c75ed1b7b9fb4819aea7fb4a97f33aec])
I20250112 02:57:42.427945 23334 tablet_loader.cc:96] loaded metadata for tablet 2f6c28220edf4598af19ba21e3e7b7d3 (table default.Table_40b78fd5c6bf456ba1ffd7b2941251ee [id=e634631ae8154ae0b29459c260451481])
I20250112 02:57:42.429572 23334 tablet_loader.cc:96] loaded metadata for tablet 334ed386f72d4fca870ade9d6a44589a (table default.Table_1b38015db69648f3a3fb3dc1c096a9e7 [id=a36fb12f741148c2964d59b1e1de32d9])
I20250112 02:57:42.447054 23334 tablet_loader.cc:96] loaded metadata for tablet 368af02998984cb2bee487b15b6a2b35 (table default.Table_3350597bcb8d4f3191e9c6ebed009ad9 [id=d84a467f72d443beb8c0af27fbc5a815])
I20250112 02:57:42.450594 23334 tablet_loader.cc:96] loaded metadata for tablet 3cb8cbaa79c84aa99bab0a0bd03bbee8 (table default.Table_94ed458009374571ae7a174587b69efc [id=87a22c25f5b74393a9ae99d46a28aaf0])
I20250112 02:57:42.467705 23334 tablet_loader.cc:96] loaded metadata for tablet 3fef7ac20f034f589e564e5f5e754705 (table default.Table_69f02882e385448497a47ae9c2398e7f [id=b66bd29a97f74532953cd91ddf226a5f])
I20250112 02:57:42.469386 23334 tablet_loader.cc:96] loaded metadata for tablet 41369773a0034c65b25fee871e6a25aa (table default.Table_03e04d34d58e463e93d16e858752d3cb [id=30ad19fdfc1246ebbeaf73fff91f925e])
I20250112 02:57:42.471156 23334 tablet_loader.cc:96] loaded metadata for tablet 43c413e1f8fe48039c4dd1b066fdd857 (table default.Table_a7196a9422b74e2db6ae571522fa7a2d [id=cd412b57888e46bba00cfdffd6cfa143])
I20250112 02:57:42.472846 23334 tablet_loader.cc:96] loaded metadata for tablet 4494e809c9ee4a0884da38b45e7a90b2 (table default.Table_ecb7cc54618b492cbaf125b216cf5ecc [id=292a711ca1cc4d1cb4b281316cbba152])
I20250112 02:57:42.474653 23334 tablet_loader.cc:96] loaded metadata for tablet 4716677cfad74c0dafe18c55796aeae5 (table default.Table_f4f19feee3254ca69d7279dcaa65cb82 [id=bcdc13e6d307474ab614c3156448492d])
I20250112 02:57:42.475710 23334 tablet_loader.cc:96] loaded metadata for tablet 486dd5218bad42b0bb9064b7a14f3a4e (table default.Table_49dc0df59ea941a5a47e56ae5218d8f4 [id=ebc9a3e7ba98471a8da862cffad452ac])
I20250112 02:57:42.476697 23334 tablet_loader.cc:96] loaded metadata for tablet 4ce1f4a9244c4b40ad8557de5b36a40e (table default.Table_be71acfdd72940f6b6f99347736902fe [id=162b28a69b4d41399328c9566221bccf])
I20250112 02:57:42.478323 23334 tablet_loader.cc:96] loaded metadata for tablet 4e4262c2fdc0440ea696e551d00b80e9 (table default.Table_f67a1fd5a8a0420db44f31cf1063241d [id=9f36fbca544045ffaf90212cf8db2ce2])
I20250112 02:57:42.479434 23334 tablet_loader.cc:96] loaded metadata for tablet 557de3de9a4d4d0bac1dc16f66ab6b01 (table default.Table_c8cd0705965747f2be723fcc60928932 [id=7edd037c5a4f48f6b511b2c5bad66ac3])
I20250112 02:57:42.480532 23334 tablet_loader.cc:96] loaded metadata for tablet 56c7f07da89c4994a57c23ebdd8061bf (table default.Table_311eae8aadc143c09b9dfa448ba3f133 [id=ab91ec4b1cad4eac8df4cbe4fb69fec2])
I20250112 02:57:42.483834 23334 tablet_loader.cc:96] loaded metadata for tablet 58c49391d2ed4c3fa84fa2d071b06094 (table default.Table_b6864a288e5f4be18484745f91242d64 [id=3522a4b6b59948b7af3eb39b0a29aa59])
I20250112 02:57:42.495934 23334 tablet_loader.cc:96] loaded metadata for tablet 5c3fd2d14dbf4bc98e42a04e33c66348 (table default.Table_8a2c2cd62fa340a1bc151dbf9a80ad8a [id=7afcc84365274d399a917883df337ac3])
I20250112 02:57:42.496932 23334 tablet_loader.cc:96] loaded metadata for tablet 5d017c05abe94657bfbd2b38329bba16 (table default.Table_f12f747999f9485880eb1238ed2ecfd9 [id=c96f30f2971d449aa4796725f41b6d8f])
I20250112 02:57:42.500304 23334 tablet_loader.cc:96] loaded metadata for tablet 62253be16a1a4b2cb7d50264e82478fc (table default.Table_933de7797e444faa9a9c4315ba2fe282 [id=ff05462c9c8d4153a8ab848c771cac12])
I20250112 02:57:42.502554 23334 tablet_loader.cc:96] loaded metadata for tablet 65eb8c99917247edbdbf5de42f3b505c (table default.Table_0d24be46f0ad49dcb9757b12d906a254 [id=a6456005326445aa9356d78a2d599e61])
I20250112 02:57:42.503666 23334 tablet_loader.cc:96] loaded metadata for tablet 66fe67a814504a89b6fa8f46e2a4aee9 (table default.Table_d17ad1e3dae4401f9292ff4a782c02e3 [id=cd1ebf2b9d2b4d428117a6d1caf38575])
I20250112 02:57:42.504748 23334 tablet_loader.cc:96] loaded metadata for tablet 67236f9e5d11467486e81393dd01e0df (table default.Table_a6fb5e40aeb440eeab4c1e01b0350539 [id=ea2c71ade61e4a8c9cb68acb9b10656e])
I20250112 02:57:42.511618 23334 tablet_loader.cc:96] loaded metadata for tablet 67260685978247898ad224128ed8964f (table default.Table_b5f650c3424f49d995b8bf5cef1f146e [id=ec58c6116f8e42c7a8f3bd8117123beb])
I20250112 02:57:42.512673 23334 tablet_loader.cc:96] loaded metadata for tablet 685953572d45476ebaf2580fd5e4a3cd (table default.Table_9456a239683340bcb3674e78ab5db82b [id=abd1389d81ad42e6a4bc4e5b7ec4e3c8])
I20250112 02:57:42.513696 23334 tablet_loader.cc:96] loaded metadata for tablet 68f5144c9c644a7f8ca95082459c4556 (table default.Table_28bd98bfe0924b169986563d8f51478c [id=0a4d2a1b809f488eaf064982567b6e8e])
I20250112 02:57:42.514807 23334 tablet_loader.cc:96] loaded metadata for tablet 698c4b2fd55d43aeb1fd5a3eaf967da5 (table default.Table_52150679084b4cd39148ad19903eebcd [id=6b7c4c86696d4ed0a68e41e4ae84e597])
I20250112 02:57:42.515867 23334 tablet_loader.cc:96] loaded metadata for tablet 6e922b68d33b4a76a260816475ee6a3c (table default.Table_6eb2a5055484422a8c96499ae9e4f9db [id=80ab88ec7dae42c39d3e35b0fbadbb3d])
I20250112 02:57:42.529983 23334 tablet_loader.cc:96] loaded metadata for tablet 750246f0a0c549229d4e7a87cd6ddfd9 (table default.Table_1ca2eecdba1a42348719978fe8a87766 [id=023cccda18f7466c8de7d483f8055740])
I20250112 02:57:42.531718 23334 tablet_loader.cc:96] loaded metadata for tablet 7634a1d8de9b495599b83f15eaa91c89 (table default.Table_ace0f644f3f149feaf59940ff5b1d217 [id=1198d5660a3444f9ba723fabeb10aff0])
I20250112 02:57:42.532716 23334 tablet_loader.cc:96] loaded metadata for tablet 779badaa22e6404aaaf1818e86228bd4 (table default.Table_e429d768998e4607aff0d49ffecbc300 [id=3e4d121aa18144cab9927e6e774fd21c])
I20250112 02:57:42.535148 23334 tablet_loader.cc:96] loaded metadata for tablet 78f03a1dfb604b33a0528c28ade287f2 (table default.Table_15c893814ebd41c2b738d9158e2ae82a [id=890880aba451487294c9107b17cf169a])
I20250112 02:57:42.536757 23334 tablet_loader.cc:96] loaded metadata for tablet 7a1144ac759a4db496f67cdc9ec35020 (table default.Table_e7b514137af3470790430ad30dcb6c5b [id=8c5090edd9e0460db5161a1909cf6bf7])
I20250112 02:57:42.537794 23334 tablet_loader.cc:96] loaded metadata for tablet 7af7e02e5fc248839b434f42f95c6509 (table default.Table_0192ff1911b647d3bb78166de4f2fc72 [id=f37e86e404294798a50a00f085825e48])
I20250112 02:57:42.538883 23334 tablet_loader.cc:96] loaded metadata for tablet 7c9b3e6c84484575acd46e61dd1ea28a (table default.Table_7992c5d054614324abbd4d5c2fc805c3 [id=0a0b7767dbad4447acfbc2981ec7d8ae])
I20250112 02:57:42.541232 23334 tablet_loader.cc:96] loaded metadata for tablet 82825c2061d74cafa814912bec005654 (table default.Table_e32d5645fc44476a881643b011a69542 [id=f6f1bd33b18440f0b4d05583eb796633])
I20250112 02:57:42.542253 23334 tablet_loader.cc:96] loaded metadata for tablet 82d2ac91c5554a64936f364e4e1978f8 (table default.Table_4150ff9a4dc84f6e979e362dbcd443b7 [id=17db4776dff34dda81c10c9fe47e9fab])
I20250112 02:57:42.543946 23334 tablet_loader.cc:96] loaded metadata for tablet 8a8d7d76e5184d668e4add9810fab7df (table default.Table_1761abbaf6d3409ab3979013e59230b0 [id=fde94eb8db8a4025b3feb72dd982f62f])
I20250112 02:57:42.545536 23334 tablet_loader.cc:96] loaded metadata for tablet 8b39a5b05b0941aea541a250bb64941e (table default.Table_6020a398e8e54211a4d2838ca624d5f8 [id=2830792215ea4aec87c5cf84095471a4])
I20250112 02:57:42.547019 23334 tablet_loader.cc:96] loaded metadata for tablet 8b6efb19f0cc4a96b957ec34c27ab836 (table default.Table_7833e03b6e984109905db04edeabfa29 [id=2cab4cc73e6145be8412cfd43b5c64a7])
I20250112 02:57:42.548367 23334 tablet_loader.cc:96] loaded metadata for tablet 929430298434492cbab3d197e1083400 (table default.Table_01f15888716948f8afeda9dc4bd2877d [id=76629e9dae0d4bbf8e7686ab7b76bfde])
I20250112 02:57:42.549396 23334 tablet_loader.cc:96] loaded metadata for tablet 95f62433f15a43eb9a654c067052cd68 (table default.Table_df360ac1d7fb44eb8e31fd606b361d07 [id=0ee497d159734165a7c6e9175ced2aeb])
I20250112 02:57:42.551702 23334 tablet_loader.cc:96] loaded metadata for tablet 9a8273b629804f1c86a813b04747a1aa (table default.Table_c7d67cd6dc514abb8a8fef6106316336 [id=9b8bea48def14faeb9293473dc2c83a4])
I20250112 02:57:42.552780 23334 tablet_loader.cc:96] loaded metadata for tablet a08141e3615244a9b266cd197d9b9ef3 (table default.Table_3dfd9d3697264cfc9249b9f772b58181 [id=d679b59fa8314116b482a5fa907d3ea4])
I20250112 02:57:42.554023 23334 tablet_loader.cc:96] loaded metadata for tablet a345af15e7664d01a184193ddb51572b (table default.Table_9e4986239f004d628ff28ec47d296db4 [id=42d4e71fcf3e4560a791f0440f61c1db])
I20250112 02:57:42.555521 23334 tablet_loader.cc:96] loaded metadata for tablet a69123c4ab384e0380c62e52b9f3b42d (table default.Table_181a7bdabe584055b44baf7ff5ab440f [id=8dbf245ab5354db7a93b54749e978c65])
I20250112 02:57:42.555943 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21359
I20250112 02:57:42.557798 23334 tablet_loader.cc:96] loaded metadata for tablet a90643c97b5e424a8bf3c262e3e5429e (table default.Table_f77d231547584322affd733f7e6103ba [id=14a6d904311440469400b09be94d6748])
I20250112 02:57:42.558779 23334 tablet_loader.cc:96] loaded metadata for tablet aab2cf8bc28b47239eaa3dcaa4c90e9c (table default.Table_484ed2b081ba48ebbe2d8c837c1cd6d8 [id=80e6455604cb4f5081be1b316c595680])
I20250112 02:57:42.559743 23334 tablet_loader.cc:96] loaded metadata for tablet ae3ed0bdc2ec44998ee461d4b1254e91 (table default.Table_33d0ee945fe34221b3848101d41218cd [id=997f3d1b76014d019ab6b75392d613b2])
I20250112 02:57:42.561239 23334 tablet_loader.cc:96] loaded metadata for tablet af0c74dd3303445b9e9aef1f2d7af62a (table default.Table_6e4197a227d046569987a0c17723f994 [id=7afacb59508a4e8abd7886d608954f4a])
I20250112 02:57:42.563774 23334 tablet_loader.cc:96] loaded metadata for tablet b0d0ba05ee4c4c9f815a4c04830b5dd4 (table default.Table_864627e87b84488693c05eabec65701d [id=8a4d6d88d54a46c4964157ad79415e0d])
I20250112 02:57:42.565624 23334 tablet_loader.cc:96] loaded metadata for tablet b32e6564d4dd4d1b9b25a14266b3dc04 (table default.Table_d173dea95e5c46a5ac100a31efa536d8 [id=8288424b5e0a46baa4471d90671cb57b])
I20250112 02:57:42.566371 23334 tablet_loader.cc:96] loaded metadata for tablet b4aff0623f504bf09011ca6de28a3b82 (table default.Table_610cf7d84db445aeab7898cdbaf84ab7 [id=e492201616514349a262e9544b133dcd])
I20250112 02:57:42.567330 23334 tablet_loader.cc:96] loaded metadata for tablet b4dddfea8e8b49189186930fce46ecc5 (table default.Table_4dc1a81866be4dad82a2dc48e27f273a [id=ded17b3c6d794d32acb8e29de164dda0])
I20250112 02:57:42.568049 23334 tablet_loader.cc:96] loaded metadata for tablet b9034065c6bc435d9f26f85a96297cd3 (table default.Table_812849daab4f4ed5b4b29a84dc3dd026 [id=e7b1f142fbd9437b9dc5da21c9095dd5])
I20250112 02:57:42.569032 23334 tablet_loader.cc:96] loaded metadata for tablet be0b4a02597842d3ae1ad0c6197cc808 (table default.Table_531aa2f73c974001b166cf2852e82831 [id=167d40ea4e0447ebbc2d269148e46041])
I20250112 02:57:42.569995 23334 tablet_loader.cc:96] loaded metadata for tablet c01590521b6e4ddc9d7fcc7142d49930 (table default.Table_3c00d9bac968444ea42965a2f7222461 [id=aab8e127fb30458a87527d74f6a098f5])
I20250112 02:57:42.571008 23334 tablet_loader.cc:96] loaded metadata for tablet c80b7813b50d4739a76776d5d16434f3 (table default.Table_4d7b917596ea4edc81355ff870e26b8e [id=e5d848f933b842389699426976918779])
I20250112 02:57:42.572050 23334 tablet_loader.cc:96] loaded metadata for tablet cf8bf172abad43ac9a519310334d1640 (table default.Table_2b8baa60a7364dc0b97fac05ac195d94 [id=ec6070ff7ece4d01ac8a745ecdf4ef0a])
I20250112 02:57:42.572775 23334 tablet_loader.cc:96] loaded metadata for tablet d1e0548276474b68919cb10fbcfcdf04 (table default.Table_3b67ac5b84644c469dcf959098497e51 [id=382beecca3e1409e9dfb4b46fc49f618])
I20250112 02:57:42.574144 23334 tablet_loader.cc:96] loaded metadata for tablet d6ca275360ec41ceb0f7ef9bfaedf647 (table default.Table_3299ba87ba964f52b97f94051b2e9431 [id=73489438d7c442c1bf2e879d10f7840f])
I20250112 02:57:42.575053 23334 tablet_loader.cc:96] loaded metadata for tablet d94cda34ff2e44cfa543362a007f1400 (table default.Table_e422009a82534de8919b09f667f83a46 [id=d23e2fba08b64c58b31aba729b015a86])
I20250112 02:57:42.575834 23334 tablet_loader.cc:96] loaded metadata for tablet da54d9dbab2f416785ec0379095bcf3e (table default.Table_6a8f570367d44d06b62d0d335dee914e [id=682416c8b9f6464789d238eb4e2a1c96])
I20250112 02:57:42.576516 23334 tablet_loader.cc:96] loaded metadata for tablet e0977a10252840a381234f02bb7339b3 (table default.Table_8324e31f290e4ea2b09e7dc7c6e76f51 [id=9864c693b5b044c3b2317b83da0b0064])
I20250112 02:57:42.577124 23334 tablet_loader.cc:96] loaded metadata for tablet e0e6beaabb6d4b099ce9e47736bb29b4 (table default.Table_e0781749d01044d5afbd6e4bb09d1c9a [id=bc51beb7f32f4576b3d2595c8d3a534c])
I20250112 02:57:42.577777 23334 tablet_loader.cc:96] loaded metadata for tablet e21a0e4b5ebc426383cf332f46492b80 (table default.Table_c43f3390842845678abdfbf57e7ff752 [id=42de45e6e7e740bba43f1d6c568c4dce])
I20250112 02:57:42.578889 23334 tablet_loader.cc:96] loaded metadata for tablet e4a9bc28dc5942608fb7224f9f058e27 (table default.Table_be69112a66094ce6871bfe8e390e087f [id=29cad3b79f5e4ff390d4c4a26e47b01d])
I20250112 02:57:42.579671 23334 tablet_loader.cc:96] loaded metadata for tablet e7d8db14a8054ae69daf42dd76f35511 (table default.Table_3ef2592713f140de9a0827e1cb98ebc4 [id=e1ba72633675415b8edec82d382ca718])
I20250112 02:57:42.580338 23334 tablet_loader.cc:96] loaded metadata for tablet e861bab8dcef41328095954a46e00a96 (table default.Table_f00f4edf62e84429be9874110b97266c [id=945572d2fec64e63bddfc5f99b89a669])
I20250112 02:57:42.581086 23334 tablet_loader.cc:96] loaded metadata for tablet e8f7301bed6c4d98a80c12b70073e62f (table default.Table_b0a5ae3947f74ee0b36a1b9bb6787c84 [id=fdcb034b8aa54ff6998c2398e3be595d])
I20250112 02:57:42.581861 23334 tablet_loader.cc:96] loaded metadata for tablet eab717cb5d794d15b9b51881b86501fc (table default.Table_54f73264193a43cba9cb43f55fa18e09 [id=2216f55937994af499e8c0cdfbf951ac])
I20250112 02:57:42.583050 23334 tablet_loader.cc:96] loaded metadata for tablet ee7967f6294b4cfeac9e97db8de41e11 (table default.Table_b36f908d431d47edbc9397abab037e2e [id=bf430ec33e1c4c8e82b19f3b9e4321b0])
I20250112 02:57:42.583791 23334 tablet_loader.cc:96] loaded metadata for tablet eef30508946e4376961b6cd463172c1f (table default.Table_9821bdb412d34479aff0378b50fc8890 [id=62417ff3d6a14befaf1f8d3e2f03bef5])
I20250112 02:57:42.584528 23334 tablet_loader.cc:96] loaded metadata for tablet f08c1affbaf247059ab65222e2e95bab (table default.Table_20d7d2b763a4431f864991b5a241d5c4 [id=71e4ca29d58a4a2782a638c3fd13bd1d])
I20250112 02:57:42.585258 23334 tablet_loader.cc:96] loaded metadata for tablet f226759c7d8b4b0d83395c542428f4c9 (table default.Table_987a7a8f641b4473bc75c11ec9f5fff9 [id=b94c4e6b8a0f4ad88683c8c06ed8ab18])
I20250112 02:57:42.586794 23334 tablet_loader.cc:96] loaded metadata for tablet f47e14720a704a19ac3aea8723e13c28 (table default.Table_7c43ca5fe80342cc8bbba92b400d35de [id=15c658ee84674bb9aae12c242ba72ae2])
I20250112 02:57:42.587749 23334 tablet_loader.cc:96] loaded metadata for tablet f663e183c2ed49b98327b2ad0de12ed9 (table default.Table_aeb5fca9ba304d2abd6c14dccdf69a16 [id=163c32c80f3640da863fd0f39a57b952])
I20250112 02:57:42.588413 23334 tablet_loader.cc:96] loaded metadata for tablet f780cf38ac4d41d8b59a3d8882a10030 (table default.Table_4ebef261df1e4307b326cfa321547410 [id=b2396e3660d1407596abd4a13e395822])
I20250112 02:57:42.597340 23334 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:57:42.614766 23334 catalog_manager.cc:1260] Loaded cluster ID: ddb158550c0a4b768edc3d530c6d0267
I20250112 02:57:42.614949 23334 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:57:42.619904 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 22265
I20250112 02:57:42.630353 23334 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:57:42.642993 23334 catalog_manager.cc:5910] T 00000000000000000000000000000000 P e76221f7d4ec44da987303d68c393a1b: Loaded TSK: 0
I20250112 02:57:42.643605 23334 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:57:42.662881 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 21504
2025-01-12T02:57:42Z chronyd exiting
I20250112 02:57:42.715111 21350 master-stress-test.cc:585] Tables created: 148
I20250112 02:57:42.715296 21350 master-stress-test.cc:586] Tables altered: 62
I20250112 02:57:42.715445 21350 master-stress-test.cc:587] Tables deleted: 55
I20250112 02:57:42.715575 21350 master-stress-test.cc:588] Tablets replaced: 22
I20250112 02:57:42.715703 21350 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (43760 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250112 02:57:42.800405 21350 test_util.cc:274] Using random seed: 2111499265
2025-01-12T02:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T02:57:42Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskxVZiJz/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskxVZiJz/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/security.properties
2025-01-12 02:57:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskxVZiJz/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskxVZiJz/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250112 02:57:56.554855 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--webserver_interface=127.20.217.190
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--hive_metastore_uris=thrift://127.0.0.1:37187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 02:57:56.857105 23741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:56.857730 23741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:56.858201 23741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:56.888676 23741 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 02:57:56.888918 23741 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 02:57:56.889171 23741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:57:56.889416 23741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:56.889660 23741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:57:56.889925 23741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:57:56.925169 23741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.217.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:56.926571 23741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:56.928190 23741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:56.945237 23750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:56.946085 23747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:57:56.946463 23748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:56.947432 23741 server_base.cc:1034] running on GCE node
I20250112 02:57:58.092118 23741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:58.095142 23741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:58.096586 23741 hybrid_clock.cc:648] HybridClock initialized: now 1736650678096538 us; error 79 us; skew 500 ppm
I20250112 02:57:58.097353 23741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:58.103444 23741 webserver.cc:458] Webserver started at http://127.20.217.190:46241/ using document root <none> and password file <none>
I20250112 02:57:58.104355 23741 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:58.104542 23741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:58.104938 23741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:58.109838 23741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/instance:
uuid: "98781ea331e8470ba27c6fc976a70be7"
format_stamp: "Formatted at 2025-01-12 02:57:58 on dist-test-slave-gppw"
I20250112 02:57:58.111025 23741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal/instance:
uuid: "98781ea331e8470ba27c6fc976a70be7"
format_stamp: "Formatted at 2025-01-12 02:57:58 on dist-test-slave-gppw"
I20250112 02:57:58.118022 23741 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250112 02:57:58.123404 23757 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:58.124365 23741 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 02:57:58.124713 23741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
uuid: "98781ea331e8470ba27c6fc976a70be7"
format_stamp: "Formatted at 2025-01-12 02:57:58 on dist-test-slave-gppw"
I20250112 02:57:58.125029 23741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:58.176028 23741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:58.177599 23741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:58.178138 23741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:58.246621 23741 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.190:33757
I20250112 02:57:58.246698 23808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.190:33757 every 8 connection(s)
I20250112 02:57:58.249362 23741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
I20250112 02:57:58.254184 23809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:58.258980 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 23741
I20250112 02:57:58.259351 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal/instance
I20250112 02:57:58.261795 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:40327
--webserver_interface=127.20.217.189
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--hive_metastore_uris=thrift://127.0.0.1:37187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:57:58.265123 23809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:58.289420 23809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:57:58.291625 23761 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:57:58.295373 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 02:57:58.344859 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 1
W20250112 02:57:58.348821 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 02:57:58.417254 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 2
W20250112 02:57:58.421284 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 02:57:58.512630 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 3
W20250112 02:57:58.516278 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:57:58.563735 23810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:57:58.564293 23810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:57:58.564715 23810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:57:58.593643 23810 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 02:57:58.593922 23810 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 02:57:58.594183 23810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:57:58.594435 23810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:57:58.594681 23810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:57:58.594933 23810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:57:58.631747 23810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:37187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.189:40327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.217.189
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:57:58.633100 23810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:57:58.634677 23810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:57:58.649909 23819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:58.659631 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 4
W20250112 02:57:58.663668 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 02:57:58.963131 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 5
W20250112 02:57:58.967417 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 02:57:59.514823 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 6
W20250112 02:57:59.521884 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:57:58.650075 23820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:58.651896 23810 server_base.cc:1034] running on GCE node
W20250112 02:57:58.651000 23822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:57:59.789803 23810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:57:59.792250 23810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:57:59.793563 23810 hybrid_clock.cc:648] HybridClock initialized: now 1736650679793525 us; error 55 us; skew 500 ppm
I20250112 02:57:59.794304 23810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:57:59.800508 23810 webserver.cc:458] Webserver started at http://127.20.217.189:45063/ using document root <none> and password file <none>
I20250112 02:57:59.801391 23810 fs_manager.cc:362] Metadata directory not provided
I20250112 02:57:59.801559 23810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:57:59.801924 23810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:57:59.806428 23810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data/instance:
uuid: "6ab49e495ba34dc29d914b277a872014"
format_stamp: "Formatted at 2025-01-12 02:57:59 on dist-test-slave-gppw"
I20250112 02:57:59.807453 23810 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal/instance:
uuid: "6ab49e495ba34dc29d914b277a872014"
format_stamp: "Formatted at 2025-01-12 02:57:59 on dist-test-slave-gppw"
I20250112 02:57:59.814173 23810 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250112 02:57:59.819118 23832 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:57:59.820148 23810 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250112 02:57:59.820420 23810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal
uuid: "6ab49e495ba34dc29d914b277a872014"
format_stamp: "Formatted at 2025-01-12 02:57:59 on dist-test-slave-gppw"
I20250112 02:57:59.820717 23810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:57:59.866670 23810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:57:59.867924 23810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:57:59.868286 23810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:57:59.934368 23810 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.189:40327
I20250112 02:57:59.934453 23883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.189:40327 every 8 connection(s)
I20250112 02:57:59.936930 23810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/data/info.pb
I20250112 02:57:59.939148 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 23810
I20250112 02:57:59.939929 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-1/wal/instance
I20250112 02:57:59.942965 23884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:57:59.943544 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.188:34169
--webserver_interface=127.20.217.188
--webserver_port=0
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--hive_metastore_uris=thrift://127.0.0.1:37187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:57:59.956197 23884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:59.978794 23884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:57:59.992503 23884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:57:59.994750 23836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 02:57:59.998478 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.047976 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 1
W20250112 02:58:00.051656 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.120139 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 2
W20250112 02:58:00.123970 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.215445 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 3
W20250112 02:58:00.219134 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 02:58:00.248313 23885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:00.249061 23885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:58:00.249789 23885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:58:00.282907 23885 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 02:58:00.283139 23885 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 02:58:00.283383 23885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:58:00.283599 23885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:58:00.283785 23885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:58:00.284003 23885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:58:00.317569 23885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:37187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.188:34169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.217.188
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:00.318848 23885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:00.320349 23885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:58:00.336460 23898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:00.362671 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 4
W20250112 02:58:00.366914 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.582487 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } attempt: 7
I20250112 02:58:00.594640 23809 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } has no permanent_uuid. Determining permanent_uuid...
W20250112 02:58:00.600414 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.658789 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 1
W20250112 02:58:00.663139 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.666349 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 5
W20250112 02:58:00.670377 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.744495 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 2
W20250112 02:58:00.748757 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.834105 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 3
W20250112 02:58:00.838305 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:00.978684 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 4
W20250112 02:58:00.982905 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:01.217753 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 6
W20250112 02:58:01.224514 23884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:01.266235 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 5
W20250112 02:58:01.270087 23809 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 02:58:00.336750 23895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:00.337661 23896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:00.338719 23885 server_base.cc:1034] running on GCE node
I20250112 02:58:01.472872 23885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:01.475302 23885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:01.476649 23885 hybrid_clock.cc:648] HybridClock initialized: now 1736650681476612 us; error 66 us; skew 500 ppm
I20250112 02:58:01.477408 23885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:01.489454 23885 webserver.cc:458] Webserver started at http://127.20.217.188:33553/ using document root <none> and password file <none>
I20250112 02:58:01.490377 23885 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:01.490602 23885 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:01.491009 23885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:58:01.495949 23885 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data/instance:
uuid: "2edd70f382cf4a44acb1db7b47eaf235"
format_stamp: "Formatted at 2025-01-12 02:58:01 on dist-test-slave-gppw"
I20250112 02:58:01.497009 23885 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal/instance:
uuid: "2edd70f382cf4a44acb1db7b47eaf235"
format_stamp: "Formatted at 2025-01-12 02:58:01 on dist-test-slave-gppw"
I20250112 02:58:01.503505 23885 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250112 02:58:01.508494 23910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:01.509440 23885 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250112 02:58:01.509742 23885 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal
uuid: "2edd70f382cf4a44acb1db7b47eaf235"
format_stamp: "Formatted at 2025-01-12 02:58:01 on dist-test-slave-gppw"
I20250112 02:58:01.510047 23885 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:01.563612 23885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:01.564985 23885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:01.565385 23885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:01.628997 23885 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.188:34169
I20250112 02:58:01.629057 23961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.188:34169 every 8 connection(s)
I20250112 02:58:01.631599 23885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/data/info.pb
I20250112 02:58:01.636112 23962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:01.639554 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 23885
I20250112 02:58:01.639946 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-2/wal/instance
I20250112 02:58:01.649780 23962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:58:01.679265 23962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:58:01.686594 23962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } has no permanent_uuid. Determining permanent_uuid...
I20250112 02:58:01.700918 23962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Bootstrap starting.
I20250112 02:58:01.706746 23962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:01.708282 23962 log.cc:826] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:01.712129 23962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: No bootstrap required, opened a new log
I20250112 02:58:01.730712 23962 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:01.731261 23962 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:01.731504 23962 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edd70f382cf4a44acb1db7b47eaf235, State: Initialized, Role: FOLLOWER
I20250112 02:58:01.732208 23962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:01.734591 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:01.735417 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:01.736256 23962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 02:58:01.766448 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.766810 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.780906 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.781328 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.759910 23885 master_runner.cc:186] Error getting master registration for 127.20.217.190:33757: OK, instance_id { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" instance_seqno: 1736650678227805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:58:01.789098 23885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:58:01.795360 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.795809 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.809758 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.810071 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.819998 23916 tablet.cc:2367] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:58:01.822530 23809 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 6
W20250112 02:58:01.823385 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.823628 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.837808 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.838121 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.850906 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.851128 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.852214 23809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap starting.
I20250112 02:58:01.860462 23809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:01.862699 23809 log.cc:826] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Log is configured to *not* fsync() on all Append() calls
W20250112 02:58:01.863824 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.864233 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.867334 23809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: No bootstrap required, opened a new log
W20250112 02:58:01.878134 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.878513 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.892762 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.893079 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.893182 23809 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:01.893828 23809 raft_consensus.cc:383] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:01.894105 23809 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98781ea331e8470ba27c6fc976a70be7, State: Initialized, Role: FOLLOWER
I20250112 02:58:01.894860 23809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:01.897650 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:01.898561 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:01.899518 23809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: configured and running, proceeding with master startup.
W20250112 02:58:01.908607 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.909080 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.924015 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.924497 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.937876 23763 tablet.cc:2367] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 02:58:01.939740 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.943917 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.958606 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.959012 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.965834 24004 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.966159 24004 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.967522 23989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:01.968027 23989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:01.970672 23989 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:01.934918 23741 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: OK, instance_id { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" instance_seqno: 1736650679915589 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250112 02:58:01.973812 23741 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:58:01.974972 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250112 02:58:01.977975 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98781ea331e8470ba27c6fc976a70be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:01.979077 23937 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98781ea331e8470ba27c6fc976a70be7 in term 0.
W20250112 02:58:01.980190 24004 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.980562 24004 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.980692 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.980561 23761 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7; no voters:
I20250112 02:58:01.981788 23989 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:01.982174 23989 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:01.982590 23989 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:01.989180 23989 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
W20250112 02:58:01.995821 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.996425 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:01.998384 24004 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:01.998706 24004 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:01.999449 23989 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 1 election: Requested vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:02.001024 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98781ea331e8470ba27c6fc976a70be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235"
I20250112 02:58:02.001569 23937 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:02.008117 23937 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98781ea331e8470ba27c6fc976a70be7 in term 1.
W20250112 02:58:02.016973 24004 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.017015 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.017279 24004 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:02.017299 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.018191 23761 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7; no voters:
I20250112 02:58:02.019006 23989 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:02.020879 23989 raft_consensus.cc:695] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 LEADER]: Becoming Leader. State: Replica: 98781ea331e8470ba27c6fc976a70be7, State: Running, Role: LEADER
I20250112 02:58:02.021796 23989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
W20250112 02:58:02.032214 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.032581 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.035007 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98781ea331e8470ba27c6fc976a70be7. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.035933 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: LEADER
W20250112 02:58:02.046501 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.046893 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250112 02:58:02.048111 24004 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.048728 24004 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.049285 24008 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:58:02.056124 24008 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250112 02:58:02.060938 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.065431 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.080531 23937 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Refusing update from remote peer 98781ea331e8470ba27c6fc976a70be7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:58:02.082186 24006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 02:58:02.097765 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.098146 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.106753 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98781ea331e8470ba27c6fc976a70be7. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.107601 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: This master's current role is: FOLLOWER
W20250112 02:58:02.112046 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.112264 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.119613 24009 mvcc.cc:204] Tried to move back new op lower bound from 7113321193786609664 to 7113321193571102720. Current Snapshot: MvccSnapshot[applied={T|T < 7113321193786609664}]
W20250112 02:58:02.128423 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.129920 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.134014 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.134805 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:02.145695 24008 catalog_manager.cc:1348] Generated new cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:02.145980 24008 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:58:02.147246 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.148010 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: LEADER
W20250112 02:58:02.152980 23984 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 02:58:02.153334 23984 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.180840 24008 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 02:58:02.186465 24008 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:58:02.194417 23984 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:02.194897 23984 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: loading cluster ID for follower catalog manager: success
I20250112 02:58:02.206279 23984 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: acquiring CA information for follower catalog manager: success
W20250112 02:58:02.212574 23984 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250112 02:58:02.226315 24008 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Generated new TSK 0
I20250112 02:58:02.229478 24008 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:58:02.236325 23984 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:58:02.239698 24008 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 02:58:02.248890 24008 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 02:58:02.265940 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.129:0
--local_ip_for_outbound_sockets=127.20.217.129
--webserver_interface=127.20.217.129
--webserver_port=0
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:58:02.285101 23884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } attempt: 7
I20250112 02:58:02.325886 23884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Bootstrap starting.
I20250112 02:58:02.339762 23884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:02.350116 23884 log.cc:826] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:02.379419 23884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: No bootstrap required, opened a new log
I20250112 02:58:02.428431 23884 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:02.429471 23884 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:02.429860 23884 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab49e495ba34dc29d914b277a872014, State: Initialized, Role: FOLLOWER
I20250112 02:58:02.430876 23884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:02.448040 23884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:58:02.448470 23857 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:02.446233 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.450660 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:02.450832 23859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98781ea331e8470ba27c6fc976a70be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:02.452663 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98781ea331e8470ba27c6fc976a70be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014"
I20250112 02:58:02.460299 23857 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Refusing update from remote peer 98781ea331e8470ba27c6fc976a70be7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 02:58:02.464129 24006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:02.536463 24017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98781ea331e8470ba27c6fc976a70be7. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.538013 24025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:02.538829 24025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:02.539382 24017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:02.601009 24037 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250112 02:58:02.643052 24037 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:02.644534 24037 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: loading cluster ID for follower catalog manager: success
I20250112 02:58:02.661219 24037 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: acquiring CA information for follower catalog manager: success
I20250112 02:58:02.678536 24037 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250112 02:58:02.874045 24014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:02.874749 24014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:58:02.875721 24014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:58:02.930989 24014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:58:02.932649 24014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.129
I20250112 02:58:02.970644 24017 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:02.971154 24017 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:02.973336 24017 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:02.975574 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:02.976890 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:02.979229 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:02.980450 24017 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:02.991822 24014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.217.129
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.129
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:02.993455 24014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:02.995455 24014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:58:03.028056 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98781ea331e8470ba27c6fc976a70be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:03.028982 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: LEADER
W20250112 02:58:03.036321 24046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:03.646883 24052 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:03.647711 24052 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:03.656818 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:03.660804 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:03.663630 23914 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235; no voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:03.666043 24053 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:03.666487 24052 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
W20250112 02:58:03.038265 24045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:03.044550 24050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:04.823253 24048 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1790 milliseconds
I20250112 02:58:04.823874 24014 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 02:58:04.825305 24014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:04.834959 24014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:04.836412 24014 hybrid_clock.cc:648] HybridClock initialized: now 1736650684836371 us; error 51 us; skew 500 ppm
I20250112 02:58:04.837512 24014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:04.857468 24014 webserver.cc:458] Webserver started at http://127.20.217.129:45081/ using document root <none> and password file <none>
I20250112 02:58:04.858983 24014 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:04.859351 24014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:04.859951 24014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:58:04.866429 24014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data/instance:
uuid: "4bea2270199d4f05ace161ff7f3fcfae"
format_stamp: "Formatted at 2025-01-12 02:58:04 on dist-test-slave-gppw"
I20250112 02:58:04.867888 24014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/instance:
uuid: "4bea2270199d4f05ace161ff7f3fcfae"
format_stamp: "Formatted at 2025-01-12 02:58:04 on dist-test-slave-gppw"
I20250112 02:58:04.883000 24014 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.014s sys 0.001s
I20250112 02:58:04.895851 24058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:04.897094 24014 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20250112 02:58:04.897481 24014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal
uuid: "4bea2270199d4f05ace161ff7f3fcfae"
format_stamp: "Formatted at 2025-01-12 02:58:04 on dist-test-slave-gppw"
I20250112 02:58:04.897922 24014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:04.927230 24059 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:04.927930 24059 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:04.938004 24059 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:04.938038 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:04.940037 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:04.941300 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:04.942066 24059 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:05.029595 24014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:05.031723 24014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:05.032279 24014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:05.048553 24014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:58:05.052698 24064 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:05.056212 24064 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:05.059072 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:05.059552 24064 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
I20250112 02:58:05.059715 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:05.061303 23914 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235; no voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:05.062356 24064 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:05.063661 24014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:58:05.064160 24014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.001s sys 0.000s
I20250112 02:58:05.064581 24014 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:58:05.064854 24014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:05.504164 24014 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.129:35241
I20250112 02:58:05.504267 24172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.129:35241 every 8 connection(s)
I20250112 02:58:05.507213 24014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/data/info.pb
I20250112 02:58:05.511684 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 24014
I20250112 02:58:05.512297 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/instance
I20250112 02:58:05.536867 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.130:0
--local_ip_for_outbound_sockets=127.20.217.130
--webserver_interface=127.20.217.130
--webserver_port=0
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:58:05.587419 24174 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:05.587998 24174 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:05.590198 24174 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:05.591209 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:05.617030 24176 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:05.617542 24176 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:05.618060 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:05.618960 24176 heartbeater.cc:510] Master 127.20.217.190:33757 requested a full tablet report, sending...
I20250112 02:58:05.619642 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:05.620813 24174 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:05.620946 24173 heartbeater.cc:346] Connected to a master server at 127.20.217.188:34169
I20250112 02:58:05.621241 24173 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:05.621960 23774 ts_manager.cc:194] Registered new tserver with Master: 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:05.622066 24173 heartbeater.cc:510] Master 127.20.217.188:34169 requested a full tablet report, sending...
I20250112 02:58:05.624599 23774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.129:40637
I20250112 02:58:05.626024 23927 ts_manager.cc:194] Registered new tserver with Master: 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:05.637781 24175 heartbeater.cc:346] Connected to a master server at 127.20.217.189:40327
I20250112 02:58:05.638130 24175 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:05.638815 24175 heartbeater.cc:510] Master 127.20.217.189:40327 requested a full tablet report, sending...
I20250112 02:58:05.641784 23849 ts_manager.cc:194] Registered new tserver with Master: 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
W20250112 02:58:06.060500 24184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:06.060997 24184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:58:06.061545 24184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:58:06.093703 24184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:58:06.094571 24184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.130
I20250112 02:58:06.128437 24176 heartbeater.cc:502] Master 127.20.217.190:33757 was elected leader, sending a full tablet report...
I20250112 02:58:06.133423 24184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.217.130
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.130
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:06.135195 24184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:06.136977 24184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:58:06.158422 24191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:06.641924 24196 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:06.642366 24196 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:06.644076 24196 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:06.645084 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:06.645215 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:06.647135 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:06.648442 24196 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:06.919008 24198 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:06.919458 24198 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:06.920857 24198 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
I20250112 02:58:06.922022 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:06.922324 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:06.923972 23913 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235; no voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:06.924531 24198 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:07.161118 24199 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:07.161482 24199 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:07.162842 24199 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:07.163822 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:07.163858 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:07.165156 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:07.165673 24199 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:58:06.158740 24190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:06.160650 24184 server_base.cc:1034] running on GCE node
W20250112 02:58:06.164362 24193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:07.349340 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:07.351594 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:07.353027 24184 hybrid_clock.cc:648] HybridClock initialized: now 1736650687352990 us; error 48 us; skew 500 ppm
I20250112 02:58:07.354022 24184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:07.366469 24184 webserver.cc:458] Webserver started at http://127.20.217.130:46157/ using document root <none> and password file <none>
I20250112 02:58:07.367460 24184 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:07.367657 24184 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:07.368117 24184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:58:07.373263 24184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data/instance:
uuid: "aff6bb94c7a14420a7540c9c198f0252"
format_stamp: "Formatted at 2025-01-12 02:58:07 on dist-test-slave-gppw"
I20250112 02:58:07.374351 24184 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/instance:
uuid: "aff6bb94c7a14420a7540c9c198f0252"
format_stamp: "Formatted at 2025-01-12 02:58:07 on dist-test-slave-gppw"
I20250112 02:58:07.381470 24184 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250112 02:58:07.386762 24204 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:07.387814 24184 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 02:58:07.388166 24184 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal
uuid: "aff6bb94c7a14420a7540c9c198f0252"
format_stamp: "Formatted at 2025-01-12 02:58:07 on dist-test-slave-gppw"
I20250112 02:58:07.388512 24184 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:07.441501 24184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:07.442974 24184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:07.443382 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:07.446367 24184 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:58:07.450614 24184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:58:07.450824 24184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:07.451031 24184 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:58:07.451210 24184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:07.589896 24184 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.130:38691
I20250112 02:58:07.590022 24316 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.130:38691 every 8 connection(s)
I20250112 02:58:07.592183 24184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/data/info.pb
I20250112 02:58:07.602265 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 24184
I20250112 02:58:07.602763 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/instance
I20250112 02:58:07.615962 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.217.131:0
--local_ip_for_outbound_sockets=127.20.217.131
--webserver_interface=127.20.217.131
--webserver_port=0
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250112 02:58:07.641443 24317 heartbeater.cc:346] Connected to a master server at 127.20.217.188:34169
I20250112 02:58:07.641932 24317 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:07.643740 24317 heartbeater.cc:510] Master 127.20.217.188:34169 requested a full tablet report, sending...
I20250112 02:58:07.644186 24319 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:07.644526 24319 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:07.645403 24319 heartbeater.cc:510] Master 127.20.217.190:33757 requested a full tablet report, sending...
I20250112 02:58:07.646656 23927 ts_manager.cc:194] Registered new tserver with Master: aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
I20250112 02:58:07.647966 23774 ts_manager.cc:194] Registered new tserver with Master: aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
I20250112 02:58:07.649566 24318 heartbeater.cc:346] Connected to a master server at 127.20.217.189:40327
I20250112 02:58:07.649857 24318 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:07.650620 24318 heartbeater.cc:510] Master 127.20.217.189:40327 requested a full tablet report, sending...
I20250112 02:58:07.651733 23774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.130:44149
I20250112 02:58:07.652946 23849 ts_manager.cc:194] Registered new tserver with Master: aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:07.939651 24327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:07.940130 24327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:58:07.940960 24327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:58:07.973615 24327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:58:07.974566 24327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.217.131
I20250112 02:58:08.009699 24327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.217.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.217.131
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.217.131
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:08.011142 24327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:08.012641 24327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:58:08.028797 24333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:08.148222 24339 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:08.148629 24339 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:08.150187 24339 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:08.151396 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:08.151333 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:08.153579 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:08.154251 24339 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:08.155319 24319 heartbeater.cc:502] Master 127.20.217.190:33757 was elected leader, sending a full tablet report...
I20250112 02:58:08.394798 24340 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:08.395249 24340 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:08.396603 24340 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
I20250112 02:58:08.397665 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:08.398036 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:08.399910 23913 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235; no voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:08.400887 24340 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:09.126654 24342 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:09.127199 24342 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:09.129222 24342 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:09.129976 23784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:09.130754 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
I20250112 02:58:09.132025 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:09.132609 24342 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:58:08.029681 24334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:08.030337 24336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:08.031426 24327 server_base.cc:1034] running on GCE node
I20250112 02:58:09.224447 24327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:09.226487 24327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:09.227895 24327 hybrid_clock.cc:648] HybridClock initialized: now 1736650689227835 us; error 66 us; skew 500 ppm
I20250112 02:58:09.228928 24327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:09.235500 24327 webserver.cc:458] Webserver started at http://127.20.217.131:35005/ using document root <none> and password file <none>
I20250112 02:58:09.236371 24327 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:09.236552 24327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:09.236968 24327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 02:58:09.243695 24327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data/instance:
uuid: "048cd781735346e9a1e53c48b25f0e38"
format_stamp: "Formatted at 2025-01-12 02:58:09 on dist-test-slave-gppw"
I20250112 02:58:09.244910 24327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/instance:
uuid: "048cd781735346e9a1e53c48b25f0e38"
format_stamp: "Formatted at 2025-01-12 02:58:09 on dist-test-slave-gppw"
I20250112 02:58:09.253690 24327 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20250112 02:58:09.260640 24347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:09.261762 24327 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250112 02:58:09.262125 24327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal
uuid: "048cd781735346e9a1e53c48b25f0e38"
format_stamp: "Formatted at 2025-01-12 02:58:09 on dist-test-slave-gppw"
I20250112 02:58:09.262559 24327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:09.330101 24327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:09.331633 24327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:09.332114 24327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:09.334481 24327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250112 02:58:09.339023 24327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 02:58:09.339215 24327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:09.339483 24327 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 02:58:09.339629 24327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:09.475687 24327 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.131:34845
I20250112 02:58:09.475796 24460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.131:34845 every 8 connection(s)
I20250112 02:58:09.478089 24327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/data/info.pb
I20250112 02:58:09.485500 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 24327
I20250112 02:58:09.486379 21350 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/instance
I20250112 02:58:09.541299 24462 heartbeater.cc:346] Connected to a master server at 127.20.217.189:40327
I20250112 02:58:09.541796 24462 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:09.541321 24461 heartbeater.cc:346] Connected to a master server at 127.20.217.188:34169
I20250112 02:58:09.542187 24461 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:09.542641 24463 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:09.542973 24463 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:09.543066 24462 heartbeater.cc:510] Master 127.20.217.189:40327 requested a full tablet report, sending...
I20250112 02:58:09.543298 24461 heartbeater.cc:510] Master 127.20.217.188:34169 requested a full tablet report, sending...
I20250112 02:58:09.543807 24463 heartbeater.cc:510] Master 127.20.217.190:33757 requested a full tablet report, sending...
I20250112 02:58:09.545719 23774 ts_manager.cc:194] Registered new tserver with Master: 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:09.545835 23848 ts_manager.cc:194] Registered new tserver with Master: 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:09.546200 23927 ts_manager.cc:194] Registered new tserver with Master: 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:09.547371 23774 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.217.131:39565
I20250112 02:58:09.563562 21350 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 02:58:09.606833 23774 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42474:
name: "default.Table_5bafe92abae84e2798e614a3069d4084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.608331 23773 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42474:
name: "default.Table_ac0f37d3057e4df2b361243d0f3e2103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.610854 23772 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42474:
name: "default.Table_9ceac5e9837d4fe99f40d0fd14a67b2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.612793 23771 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42474:
name: "default.Table_51ffd08ccf99400ba427786f23d0bcfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:58:09.618180 23773 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac0f37d3057e4df2b361243d0f3e2103 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:09.624315 23774 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5bafe92abae84e2798e614a3069d4084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:09.635393 23772 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:09.635430 23771 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51ffd08ccf99400ba427786f23d0bcfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:58:09.650668 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 23741
I20250112 02:58:09.667188 24495 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:09.668789 24495 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:09.671309 24495 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:09.672230 23937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2edd70f382cf4a44acb1db7b47eaf235" is_pre_election: true
W20250112 02:58:09.693459 24062 connection.cc:537] client connection to 127.20.217.190:33757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:09.693450 24207 connection.cc:537] client connection to 127.20.217.190:33757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:09.694175 23835 connection.cc:537] client connection to 127.20.217.190:33757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:09.694283 24176 heartbeater.cc:643] Failed to heartbeat to 127.20.217.190:33757 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:09.694273 24319 heartbeater.cc:643] Failed to heartbeat to 127.20.217.190:33757 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:09.694614 23835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 6 similar messages]
W20250112 02:58:09.709275 23835 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
I20250112 02:58:09.709838 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:09.711194 24495 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250112 02:58:09.694840 24473 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.217.190:33757)
W20250112 02:58:09.762770 24473 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.20.217.190:33757, user_credentials={real_user=slave}} blocked reactor thread for 69924.6us
I20250112 02:58:09.823168 24498 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:09.823632 24498 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:09.830392 24498 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
I20250112 02:58:09.831276 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:09.831985 23858 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edd70f382cf4a44acb1db7b47eaf235 in term 1.
I20250112 02:58:09.834338 23914 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014; no voters:
I20250112 02:58:09.835876 24498 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 02:58:09.836262 24498 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 98781ea331e8470ba27c6fc976a70be7)
I20250112 02:58:09.836686 24498 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 1 FOLLOWER]: Advancing to term 2
W20250112 02:58:09.844769 23913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
I20250112 02:58:09.848405 24498 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:09.860312 23858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2edd70f382cf4a44acb1db7b47eaf235" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014"
I20250112 02:58:09.860881 23858 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 1 FOLLOWER]: Advancing to term 2
W20250112 02:58:09.871860 23913 leader_election.cc:336] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
I20250112 02:58:09.873279 23858 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edd70f382cf4a44acb1db7b47eaf235 in term 2.
I20250112 02:58:09.874406 24498 leader_election.cc:290] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 election: Requested vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327)
W20250112 02:58:09.876672 23913 leader_election.cc:336] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
I20250112 02:58:09.878262 23914 leader_election.cc:304] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014; no voters: 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:09.879323 24498 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 02:58:09.890966 24498 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [term 2 LEADER]: Becoming Leader. State: Replica: 2edd70f382cf4a44acb1db7b47eaf235, State: Running, Role: LEADER
I20250112 02:58:09.891975 24498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:09.900278 24501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edd70f382cf4a44acb1db7b47eaf235. Latest consensus state: current_term: 2 leader_uuid: "2edd70f382cf4a44acb1db7b47eaf235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:09.901124 24501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: This master's current role is: LEADER
I20250112 02:58:09.905745 24503 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:58:09.914052 24503 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:58:09.923416 24503 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:09.923768 24503 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:58:09.927915 24503 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:58:09.932081 24503 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Loaded TSK: 0
I20250112 02:58:09.934662 24503 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:58:09.942305 24503 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 02:58:09.951699 24503 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 02:58:09.963609 23927 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_51ffd08ccf99400ba427786f23d0bcfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.968261 23926 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_9ceac5e9837d4fe99f40d0fd14a67b2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.969755 23925 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_ac0f37d3057e4df2b361243d0f3e2103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:09.971340 23924 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_5bafe92abae84e2798e614a3069d4084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:58:09.974432 23927 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_51ffd08ccf99400ba427786f23d0bcfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:09.982244 23926 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:09.984052 23925 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ac0f37d3057e4df2b361243d0f3e2103 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250112 02:58:10.001942 23924 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_5bafe92abae84e2798e614a3069d4084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 02:58:10.049865 24463 heartbeater.cc:502] Master 127.20.217.190:33757 was elected leader, sending a full tablet report...
W20250112 02:58:10.055130 24463 heartbeater.cc:643] Failed to heartbeat to 127.20.217.190:33757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
I20250112 02:58:10.214107 23927 catalog_manager.cc:2149] created HMS catalog entry for table [id=814d533e9fb741cab793d3e6a98ee15e]
I20250112 02:58:10.219496 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--webserver_interface=127.20.217.190
--webserver_port=46241
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--hive_metastore_uris=thrift://127.0.0.1:37187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250112 02:58:10.222165 23858 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Refusing update from remote peer 2edd70f382cf4a44acb1db7b47eaf235: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250112 02:58:10.223950 24501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250112 02:58:10.230190 23913 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 -> Peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Couldn't send request to peer 98781ea331e8470ba27c6fc976a70be7. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 02:58:10.267122 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edd70f382cf4a44acb1db7b47eaf235. Latest consensus state: current_term: 2 leader_uuid: "2edd70f382cf4a44acb1db7b47eaf235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:10.268041 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:10.283730 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2edd70f382cf4a44acb1db7b47eaf235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:10.284489 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:10.294198 24498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2edd70f382cf4a44acb1db7b47eaf235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:10.295354 24498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235 [sys.catalog]: This master's current role is: LEADER
I20250112 02:58:10.304363 23927 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41404:
table { table_name: "default.Table_51ffd08ccf99400ba427786f23d0bcfd" } new_table_name: "default.Table_a9f5d2a7adb243439a6be2d0b74d48c0" modify_external_catalogs: true
I20250112 02:58:10.371681 24108 tablet_service.cc:1467] Processing CreateTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.373538 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c498b953cdbf4f4abcf3eec9b385ea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.375774 24395 tablet_service.cc:1467] Processing CreateTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.377564 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c498b953cdbf4f4abcf3eec9b385ea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.390753 24252 tablet_service.cc:1467] Processing CreateTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.392612 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c498b953cdbf4f4abcf3eec9b385ea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.425835 24519 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:10.444087 24520 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:10.458784 24521 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:10.458644 23923 master_service.cc:946] ReplaceTablet: received request to replace tablet c498b953cdbf4f4abcf3eec9b385ea83 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:10.464679 23922 master_service.cc:946] ReplaceTablet: received request to replace tablet c498b953cdbf4f4abcf3eec9b385ea83 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:10.470199 24520 tablet_bootstrap.cc:654] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:10.475009 24519 tablet_bootstrap.cc:654] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:10.477250 24519 log.cc:826] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:10.480134 24521 tablet_bootstrap.cc:654] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:10.482547 24521 log.cc:826] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:10.489976 24519 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:10.490573 24519 ts_tablet_manager.cc:1397] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.007s
I20250112 02:58:10.490723 23923 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet c498b953cdbf4f4abcf3eec9b385ea83
I20250112 02:58:10.486476 24520 log.cc:826] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:10.493826 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:10.494197 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.492655 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.511062 23922 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_76f3bea64f5b4d869d3c8ec3ef52189e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:10.527058 24107 tablet_service.cc:1467] Processing CreateTablet for tablet a1da6a48f052437180cd76d468e8eea2 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.528321 24107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6a48f052437180cd76d468e8eea2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:58:10.529259 23911 catalog_manager.cc:4852] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): delete failed for tablet c498b953cdbf4f4abcf3eec9b385ea83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c498b953cdbf4f4abcf3eec9b385ea83 already in progress: creating tablet
I20250112 02:58:10.533874 24520 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
W20250112 02:58:10.536484 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet c498b953cdbf4f4abcf3eec9b385ea83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c498b953cdbf4f4abcf3eec9b385ea83 already in progress: creating tablet
W20250112 02:58:10.537945 23913 catalog_manager.cc:4852] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): delete failed for tablet c498b953cdbf4f4abcf3eec9b385ea83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c498b953cdbf4f4abcf3eec9b385ea83 already in progress: creating tablet
I20250112 02:58:10.534426 24520 ts_tablet_manager.cc:1397] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.099s user 0.005s sys 0.009s
I20250112 02:58:10.539736 23923 catalog_manager.cc:6596] ReplaceTablet: tablet c498b953cdbf4f4abcf3eec9b385ea83 deleted and replaced by tablet a1da6a48f052437180cd76d468e8eea2
I20250112 02:58:10.546556 24252 tablet_service.cc:1467] Processing CreateTablet for tablet a1da6a48f052437180cd76d468e8eea2 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.547971 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6a48f052437180cd76d468e8eea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.553669 24461 heartbeater.cc:502] Master 127.20.217.188:34169 was elected leader, sending a full tablet report...
I20250112 02:58:10.567256 24395 tablet_service.cc:1467] Processing CreateTablet for tablet a1da6a48f052437180cd76d468e8eea2 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.568384 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1da6a48f052437180cd76d468e8eea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.575227 24394 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:40628
W20250112 02:58:10.576844 23913 catalog_manager.cc:4852] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): delete failed for tablet c498b953cdbf4f4abcf3eec9b385ea83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c498b953cdbf4f4abcf3eec9b385ea83 already in progress: creating tablet
I20250112 02:58:10.589488 24521 tablet_bootstrap.cc:492] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:10.590905 24521 ts_tablet_manager.cc:1397] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.139s user 0.026s sys 0.018s
I20250112 02:58:10.619220 24519 raft_consensus.cc:357] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.620306 24519 raft_consensus.cc:383] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.620656 24519 raft_consensus.cc:738] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.621707 24519 consensus_queue.cc:260] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.631793 24519 ts_tablet_manager.cc:1428] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.141s user 0.040s sys 0.000s
I20250112 02:58:10.639286 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:10.640121 24524 tablet_replica.cc:331] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:10.640825 24524 raft_consensus.cc:2238] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.641362 24524 raft_consensus.cc:2267] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.651997 24519 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:10.673223 24524 ts_tablet_manager.cc:1905] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:10.673842 24521 raft_consensus.cc:357] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.675107 24521 raft_consensus.cc:383] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.675449 24521 raft_consensus.cc:738] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.675268 24520 raft_consensus.cc:357] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.676172 24520 raft_consensus.cc:383] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.676440 24520 raft_consensus.cc:738] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.677302 24520 consensus_queue.cc:260] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.678306 24521 consensus_queue.cc:260] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } }
I20250112 02:58:10.682178 24317 heartbeater.cc:502] Master 127.20.217.188:34169 was elected leader, sending a full tablet report...
02:58:10.664 [HMS - ERROR - pool-6-thread-1] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_ac0f37d3057e4df2b361243d0f3e2103 already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I20250112 02:58:10.683627 24173 heartbeater.cc:502] Master 127.20.217.188:34169 was elected leader, sending a full tablet report...
I20250112 02:58:10.686939 24519 tablet_bootstrap.cc:654] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:10.687817 23925 catalog_manager.cc:2145] Already present: failed to create HMS catalog entry for table [id=07e101dc6d3843fe9d3ca180d56a11bd]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_ac0f37d3057e4df2b361243d0f3e2103 already exists)
I20250112 02:58:10.699738 23923 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_ebcfbc2fbb6442918a4444cc0bcbab4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:10.702574 24520 ts_tablet_manager.cc:1428] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.163s user 0.027s sys 0.010s
I20250112 02:58:10.703459 24520 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:10.705780 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet a1da6a48f052437180cd76d468e8eea2 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:10.710001 24520 tablet_bootstrap.cc:654] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:10.731568 24521 ts_tablet_manager.cc:1428] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.140s user 0.035s sys 0.001s
I20250112 02:58:10.737893 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.738829 24525 tablet_replica.cc:331] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:10.739596 24521 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:10.739648 24525 raft_consensus.cc:2238] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.742075 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet c498b953cdbf4f4abcf3eec9b385ea83 with delete_type TABLET_DATA_DELETED (replaced by tablet a1da6a48f052437180cd76d468e8eea2) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.742578 24525 raft_consensus.cc:2267] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.742586 23920 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet a1da6a48f052437180cd76d468e8eea2
I20250112 02:58:10.743332 24526 tablet_replica.cc:331] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:10.744128 24526 raft_consensus.cc:2238] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.744652 24526 raft_consensus.cc:2267] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.746042 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.747202 23920 catalog_manager.cc:6596] ReplaceTablet: tablet a1da6a48f052437180cd76d468e8eea2 deleted and replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:10.747771 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:10.748924 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.749297 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.749990 24526 ts_tablet_manager.cc:1905] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:10.751551 24525 ts_tablet_manager.cc:1905] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:10.751739 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.758742 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet a1da6a48f052437180cd76d468e8eea2 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:10.774904 24524 ts_tablet_manager.cc:1918] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:10.775346 24524 log.cc:1198] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/c498b953cdbf4f4abcf3eec9b385ea83
I20250112 02:58:10.776214 24524 ts_tablet_manager.cc:1939] T c498b953cdbf4f4abcf3eec9b385ea83 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:10.782822 24525 ts_tablet_manager.cc:1918] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:10.795079 24525 log.cc:1198] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/c498b953cdbf4f4abcf3eec9b385ea83
I20250112 02:58:10.796054 24525 ts_tablet_manager.cc:1939] T c498b953cdbf4f4abcf3eec9b385ea83 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:10.798943 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet c498b953cdbf4f4abcf3eec9b385ea83 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:10.799333 24521 tablet_bootstrap.cc:654] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:10.799938 23911 catalog_manager.cc:4852] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
W20250112 02:58:10.800717 23911 catalog_manager.cc:4852] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
I20250112 02:58:10.806097 24519 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:10.807348 24519 ts_tablet_manager.cc:1397] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.156s user 0.008s sys 0.004s
I20250112 02:58:10.810925 24519 raft_consensus.cc:357] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.816113 24519 raft_consensus.cc:383] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.816111 24108 tablet_service.cc:1467] Processing CreateTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.817138 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c6df7f45f04d46b788c13f1c3f2455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.816978 24519 raft_consensus.cc:738] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.818708 24526 ts_tablet_manager.cc:1918] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:10.819111 24526 log.cc:1198] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/c498b953cdbf4f4abcf3eec9b385ea83
I20250112 02:58:10.820060 24526 ts_tablet_manager.cc:1939] T c498b953cdbf4f4abcf3eec9b385ea83 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:10.823326 23912 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet c498b953cdbf4f4abcf3eec9b385ea83 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:10.827476 24521 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:10.827930 24521 ts_tablet_manager.cc:1397] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.002s
I20250112 02:58:10.827631 24252 tablet_service.cc:1467] Processing CreateTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.829154 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c6df7f45f04d46b788c13f1c3f2455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.830420 24521 raft_consensus.cc:357] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
W20250112 02:58:10.834890 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
W20250112 02:58:10.835857 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
I20250112 02:58:10.839468 24521 raft_consensus.cc:383] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.839828 24521 raft_consensus.cc:738] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.840646 24521 consensus_queue.cc:260] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.817821 24519 consensus_queue.cc:260] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.855954 24521 ts_tablet_manager.cc:1428] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250112 02:58:10.862536 24519 ts_tablet_manager.cc:1428] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
W20250112 02:58:10.864001 23913 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.217.190:33757, user_credentials={real_user=slave}} blocked reactor thread for 70288.7us
I20250112 02:58:10.865059 24529 raft_consensus.cc:491] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250112 02:58:10.866178 24322 tablet.cc:2367] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:58:10.866080 24395 tablet_service.cc:1467] Processing CreateTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:10.865717 24529 raft_consensus.cc:513] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.867172 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6c6df7f45f04d46b788c13f1c3f2455. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:10.867733 23913 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet c498b953cdbf4f4abcf3eec9b385ea83 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
W20250112 02:58:10.868420 23913 catalog_manager.cc:4852] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
I20250112 02:58:10.872593 24529 leader_election.cc:290] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
I20250112 02:58:10.878981 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.879727 24525 tablet_replica.cc:331] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:10.880506 24525 raft_consensus.cc:2238] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.880906 24525 raft_consensus.cc:2267] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.883282 24525 ts_tablet_manager.cc:1905] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:10.888655 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1da6a48f052437180cd76d468e8eea2" candidate_uuid: "048cd781735346e9a1e53c48b25f0e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252" is_pre_election: true
W20250112 02:58:10.890638 24348 leader_election.cc:343] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:58:10.894392 24520 tablet_bootstrap.cc:492] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:10.901511 24520 ts_tablet_manager.cc:1397] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.198s user 0.007s sys 0.004s
I20250112 02:58:10.905225 24521 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:10.905692 24520 raft_consensus.cc:357] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.906395 24520 raft_consensus.cc:383] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.906685 24520 raft_consensus.cc:738] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:10.907452 24520 consensus_queue.cc:260] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.912675 24521 tablet_bootstrap.cc:654] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:10.918884 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:53670
W20250112 02:58:10.920650 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet a1da6a48f052437180cd76d468e8eea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1da6a48f052437180cd76d468e8eea2 already in progress: creating tablet
I20250112 02:58:10.925470 24520 ts_tablet_manager.cc:1428] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.024s user 0.000s sys 0.006s
I20250112 02:58:10.945577 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.946437 24526 tablet_replica.cc:331] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:10.947263 24526 raft_consensus.cc:2238] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.947786 24526 raft_consensus.cc:2267] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.951339 24526 ts_tablet_manager.cc:1905] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:10.953750 23921 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:10.964293 24394 tablet_service.cc:1514] Processing DeleteTablet for tablet a1da6a48f052437180cd76d468e8eea2 with delete_type TABLET_DATA_DELETED (replaced by tablet b6c6df7f45f04d46b788c13f1c3f2455) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:10.966087 24521 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:10.966477 24521 ts_tablet_manager.cc:1397] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.016s
I20250112 02:58:10.967438 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1da6a48f052437180cd76d468e8eea2" candidate_uuid: "048cd781735346e9a1e53c48b25f0e38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae" is_pre_election: true
W20250112 02:58:10.968981 24349 leader_election.cc:343] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Illegal state: Tablet not RUNNING: STOPPED
I20250112 02:58:10.969432 24349 leader_election.cc:304] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:10.969846 24524 tablet_replica.cc:331] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:10.970350 24529 raft_consensus.cc:2743] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:10.971999 24521 raft_consensus.cc:357] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.972683 24521 raft_consensus.cc:383] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:10.972939 24521 raft_consensus.cc:738] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
W20250112 02:58:10.973198 24510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:10.973964 24510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20250112 02:58:10.973579 24521 consensus_queue.cc:260] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:10.976140 24521 ts_tablet_manager.cc:1428] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250112 02:58:10.976476 23925 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
W20250112 02:58:10.974725 24510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250112 02:58:10.979619 24524 raft_consensus.cc:2238] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.980690 24525 ts_tablet_manager.cc:1918] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:10.981071 24525 log.cc:1198] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/a1da6a48f052437180cd76d468e8eea2
I20250112 02:58:10.981676 24525 ts_tablet_manager.cc:1939] T a1da6a48f052437180cd76d468e8eea2 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:10.981642 24519 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:10.987236 23921 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:10.988548 23921 catalog_manager.cc:6596] ReplaceTablet: tablet b6c6df7f45f04d46b788c13f1c3f2455 deleted and replaced by tablet d5a63ecb8eed451a8027d28bd2e8d91f
I20250112 02:58:10.990370 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 with delete_type TABLET_DATA_DELETED (replaced by tablet d5a63ecb8eed451a8027d28bd2e8d91f) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:10.991070 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 with delete_type TABLET_DATA_DELETED (replaced by tablet d5a63ecb8eed451a8027d28bd2e8d91f) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:10.992630 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet b6c6df7f45f04d46b788c13f1c3f2455 with delete_type TABLET_DATA_DELETED (replaced by tablet d5a63ecb8eed451a8027d28bd2e8d91f) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:10.997579 24525 tablet_replica.cc:331] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:10.998248 24525 raft_consensus.cc:2238] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:10.998705 24525 raft_consensus.cc:2267] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:10.998834 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet a1da6a48f052437180cd76d468e8eea2 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:11.001060 24525 ts_tablet_manager.cc:1905] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.006270 24519 tablet_bootstrap.cc:654] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.011080 24524 raft_consensus.cc:2267] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.013643 24524 ts_tablet_manager.cc:1905] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.014480 24525 ts_tablet_manager.cc:1918] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.014930 24525 log.cc:1198] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.015635 24525 ts_tablet_manager.cc:1939] T b6c6df7f45f04d46b788c13f1c3f2455 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:11.018067 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet b6c6df7f45f04d46b788c13f1c3f2455 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:11.021929 24526 ts_tablet_manager.cc:1918] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.023407 24252 tablet_service.cc:1467] Processing CreateTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.024413 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a63ecb8eed451a8027d28bd2e8d91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.022488 24108 tablet_service.cc:1467] Processing CreateTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.031893 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a63ecb8eed451a8027d28bd2e8d91f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250112 02:58:11.035599 24510 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 02:58:11.035890 24510 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 02:58:11.036181 24510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:58:11.036480 24510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20250112 02:58:11.036098 24395 tablet_service.cc:1467] Processing CreateTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250112 02:58:11.036784 24510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:58:11.037083 24510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:58:11.037454 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a63ecb8eed451a8027d28bd2e8d91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.026222 24526 log.cc:1198] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/a1da6a48f052437180cd76d468e8eea2
I20250112 02:58:11.054139 24526 ts_tablet_manager.cc:1939] T a1da6a48f052437180cd76d468e8eea2 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:11.056732 23912 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet a1da6a48f052437180cd76d468e8eea2 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
W20250112 02:58:11.057837 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet b6c6df7f45f04d46b788c13f1c3f2455 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6c6df7f45f04d46b788c13f1c3f2455 already in progress: creating tablet
I20250112 02:58:11.059275 24519 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:11.052904 24520 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:11.060468 24519 ts_tablet_manager.cc:1397] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.032s
I20250112 02:58:11.062947 24519 raft_consensus.cc:357] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.074324 24524 ts_tablet_manager.cc:1918] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.074728 24524 log.cc:1198] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/a1da6a48f052437180cd76d468e8eea2
I20250112 02:58:11.075328 24524 ts_tablet_manager.cc:1939] T a1da6a48f052437180cd76d468e8eea2 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:11.076826 24519 raft_consensus.cc:383] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.077101 24519 raft_consensus.cc:738] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.077770 24519 consensus_queue.cc:260] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.083992 24520 tablet_bootstrap.cc:654] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.086053 24519 ts_tablet_manager.cc:1428] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250112 02:58:11.087366 23913 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet a1da6a48f052437180cd76d468e8eea2 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:11.088239 24524 tablet_replica.cc:331] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:11.088830 24524 raft_consensus.cc:2238] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.089285 24524 raft_consensus.cc:2267] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.090848 23926 catalog_manager.cc:2149] created HMS catalog entry for table [id=0f37e561467548519e44e122c0fe2d0a]
I20250112 02:58:11.099646 24510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.217.190
--webserver_port=46241
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:11.100432 24519 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:11.101424 24510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:11.103569 24510 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 02:58:11.104678 24521 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:11.105778 24524 ts_tablet_manager.cc:1905] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250112 02:58:11.106155 23925 catalog_manager.cc:5359] Ignoring report from unknown tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.110399 24521 tablet_bootstrap.cc:654] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:11.112116 23925 catalog_manager.cc:5359] Ignoring report from unknown tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.133428 24519 tablet_bootstrap.cc:654] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.136142 23926 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41404 (request call id 14) took 1169 ms. Trace:
W20250112 02:58:11.143599 24550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:11.149704 24524 ts_tablet_manager.cc:1918] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.150146 24524 log.cc:1198] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.150868 24524 ts_tablet_manager.cc:1939] T b6c6df7f45f04d46b788c13f1c3f2455 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:11.136493 23926 rpcz_store.cc:276] 0112 02:58:09.966895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:58:09.967682 (+ 787us) service_pool.cc:225] Handling call
0112 02:58:09.982685 (+ 15003us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:58:09.999529 (+ 16844us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:58:11.090693 (+1091164us) catalog_manager.cc:2148] Created new table in HMS catalog
0112 02:58:11.135423 (+ 44730us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:58:11.135744 (+ 321us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:58:11.136078 (+ 334us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:58:11.093338 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:58:11.093752 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:58:11.093775 (+ 23us) write_op.cc:435] Acquired schema lock
0112 02:58:11.093791 (+ 16us) tablet.cc:659] Decoding operations
0112 02:58:11.094070 (+ 279us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:58:11.094108 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0112 02:58:11.094132 (+ 24us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:58:11.094342 (+ 210us) tablet.cc:698] Row locks acquired
0112 02:58:11.094358 (+ 16us) write_op.cc:260] PREPARE: finished
0112 02:58:11.094472 (+ 114us) write_op.cc:270] Start()
0112 02:58:11.095006 (+ 534us) write_op.cc:276] Timestamp: P: 1736650691094459 usec, L: 0
0112 02:58:11.095024 (+ 18us) op_driver.cc:348] REPLICATION: starting
0112 02:58:11.118292 (+ 23268us) log.cc:844] Serialized 432 byte log entry
0112 02:58:11.124626 (+ 6334us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b54000415d4. stack: 00007f172d13f853 00007f172d13f750 00007f172ccb4584 00000000004fcd24 00000000004fccb9 00007f173403f6d1 00007f17340b6198 00007f17345223b1 00007f1734521b88 00007f1734526821 00007f17345267d9 00007f1734526769 00007f1734526731 00007f173452595d 000000000059ea04 000000000059e839
0112 02:58:11.133058 (+ 8432us) op_driver.cc:464] REPLICATION: finished
0112 02:58:11.133442 (+ 384us) write_op.cc:301] APPLY: starting
0112 02:58:11.133496 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0112 02:58:11.133595 (+ 99us) tablet.cc:1367] finished BulkCheckPresence
0112 02:58:11.133612 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:58:11.133954 (+ 342us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:58:11.134469 (+ 515us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:58:11.134531 (+ 62us) write_op.cc:312] APPLY: finished
0112 02:58:11.134718 (+ 187us) log.cc:844] Serialized 28 byte log entry
0112 02:58:11.134855 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:58:11.134933 (+ 78us) write_op.cc:454] Released schema lock
0112 02:58:11.134965 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0112 02:58:11.135165 (+ 200us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:58:11.135187 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":686033,"Hive Metastore.run_cpu_time_us":650,"Hive Metastore.run_wall_time_us":404557,"child_traces":[["op",{"apply.queue_time_us":241,"apply.run_cpu_time_us":1340,"apply.run_wall_time_us":1694,"num_ops":2,"prepare.queue_time_us":252,"prepare.run_cpu_time_us":4265,"prepare.run_wall_time_us":31567,"raft.queue_time_us":7503,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":14,"replication_time_us":37950,"spinlock_wait_cycles":51860736}]]}
I20250112 02:58:11.157081 23926 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41404:
table { table_name: "default.Table_9ceac5e9837d4fe99f40d0fd14a67b2b" } new_table_name: "default.Table_f1941917eac84909bd5258d5518637b6" modify_external_catalogs: true
I20250112 02:58:11.161264 24520 tablet_bootstrap.cc:492] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.161386 24519 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:11.161688 24520 ts_tablet_manager.cc:1397] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.109s user 0.007s sys 0.011s
I20250112 02:58:11.161824 24519 ts_tablet_manager.cc:1397] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.004s
I20250112 02:58:11.165056 24519 raft_consensus.cc:357] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.165946 24519 raft_consensus.cc:383] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.166270 24519 raft_consensus.cc:738] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.167080 24519 consensus_queue.cc:260] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.164384 24520 raft_consensus.cc:357] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.176692 23913 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet b6c6df7f45f04d46b788c13f1c3f2455 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:11.177919 24519 ts_tablet_manager.cc:1428] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20250112 02:58:11.179296 24520 raft_consensus.cc:383] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.179596 24520 raft_consensus.cc:738] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.188500 24520 consensus_queue.cc:260] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.192930 24520 ts_tablet_manager.cc:1428] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250112 02:58:11.193830 24520 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
W20250112 02:58:11.195823 23920 catalog_manager.cc:5359] Ignoring report from unknown tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.203583 23921 master_service.cc:946] ReplaceTablet: received request to replace tablet d5a63ecb8eed451a8027d28bd2e8d91f from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.206065 24520 tablet_bootstrap.cc:654] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:11.207504 23919 catalog_manager.cc:5359] Ignoring report from unknown tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.210574 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet d5a63ecb8eed451a8027d28bd2e8d91f from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.224851 24252 tablet_service.cc:1467] Processing CreateTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.225934 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a91ada2bb547ceaf7b06a6a6764c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.229753 24108 tablet_service.cc:1467] Processing CreateTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.232270 24395 tablet_service.cc:1467] Processing CreateTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.232640 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a91ada2bb547ceaf7b06a6a6764c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.233295 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a91ada2bb547ceaf7b06a6a6764c74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.239591 24521 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:11.240027 24521 ts_tablet_manager.cc:1397] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.136s user 0.011s sys 0.007s
I20250112 02:58:11.242712 24521 raft_consensus.cc:357] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.243394 24521 raft_consensus.cc:383] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.243671 24521 raft_consensus.cc:738] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.244366 24521 consensus_queue.cc:260] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.246768 24521 ts_tablet_manager.cc:1428] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20250112 02:58:11.273838 23921 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet d5a63ecb8eed451a8027d28bd2e8d91f
I20250112 02:58:11.275290 23921 catalog_manager.cc:6596] ReplaceTablet: tablet d5a63ecb8eed451a8027d28bd2e8d91f deleted and replaced by tablet fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:11.276278 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f with delete_type TABLET_DATA_DELETED (replaced by tablet fba5ffea5d5f4bf1a734eccee2cfc3c8) from {username='slave'} at 127.0.0.1:47790
W20250112 02:58:11.276913 24468 tablet.cc:2367] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:58:11.278177 24394 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f with delete_type TABLET_DATA_DELETED (replaced by tablet fba5ffea5d5f4bf1a734eccee2cfc3c8) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:11.278852 24525 tablet_replica.cc:331] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:11.279498 24525 raft_consensus.cc:2238] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.279956 24525 raft_consensus.cc:2267] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.281013 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f with delete_type TABLET_DATA_DELETED (replaced by tablet fba5ffea5d5f4bf1a734eccee2cfc3c8) from {username='slave'} at 127.0.0.1:53670
W20250112 02:58:11.283442 23912 catalog_manager.cc:4852] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): delete failed for tablet d5a63ecb8eed451a8027d28bd2e8d91f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d5a63ecb8eed451a8027d28bd2e8d91f already in progress: creating tablet
I20250112 02:58:11.284129 24524 tablet_replica.cc:331] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:11.284770 24524 raft_consensus.cc:2238] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.285228 24524 raft_consensus.cc:2267] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.288403 24524 ts_tablet_manager.cc:1905] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.291899 24525 ts_tablet_manager.cc:1905] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.296500 24519 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:11.304067 23924 catalog_manager.cc:2149] created HMS catalog entry for table [id=cba9cafb990f42bfbdaa83dc8379d52a]
I20250112 02:58:11.304729 24530 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:11.305269 24530 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.308625 24530 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:11.310964 24519 tablet_bootstrap.cc:654] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:11.325631 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:11.327147 24181 tablet.cc:2367] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 02:58:11.331817 24521 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:11.334576 24519 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:11.335147 24519 ts_tablet_manager.cc:1397] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.000s
I20250112 02:58:11.338486 24519 raft_consensus.cc:357] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.339536 24519 raft_consensus.cc:383] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.339936 24519 raft_consensus.cc:738] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.342123 24521 tablet_bootstrap.cc:654] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.342451 24519 consensus_queue.cc:260] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.344550 24519 ts_tablet_manager.cc:1428] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20250112 02:58:11.346767 23924 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:41404 (request call id 16) took 1379 ms. Trace:
I20250112 02:58:11.347187 23924 rpcz_store.cc:276] 0112 02:58:09.967554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 02:58:09.970788 (+ 3234us) service_pool.cc:225] Handling call
0112 02:58:10.002406 (+ 31618us) catalog_manager.cc:2076] Acquired catalog manager lock
0112 02:58:10.004157 (+ 1751us) catalog_manager.cc:2122] Created new table and tablet info
0112 02:58:11.303883 (+1299726us) catalog_manager.cc:2148] Created new table in HMS catalog
0112 02:58:11.345810 (+ 41927us) catalog_manager.cc:2182] Wrote table and tablets to system table
0112 02:58:11.346277 (+ 467us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0112 02:58:11.346694 (+ 417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 02:58:11.306864 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0112 02:58:11.307436 (+ 572us) write_op.cc:432] Acquiring schema lock in shared mode
0112 02:58:11.307472 (+ 36us) write_op.cc:435] Acquired schema lock
0112 02:58:11.307491 (+ 19us) tablet.cc:659] Decoding operations
0112 02:58:11.307880 (+ 389us) write_op.cc:620] Acquiring the partition lock for write op
0112 02:58:11.307951 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0112 02:58:11.307987 (+ 36us) tablet.cc:682] Acquiring locks for 2 operations
0112 02:58:11.311413 (+ 3426us) tablet.cc:698] Row locks acquired
0112 02:58:11.311436 (+ 23us) write_op.cc:260] PREPARE: finished
0112 02:58:11.311573 (+ 137us) write_op.cc:270] Start()
0112 02:58:11.311671 (+ 98us) write_op.cc:276] Timestamp: P: 1736650691311525 usec, L: 0
0112 02:58:11.311687 (+ 16us) op_driver.cc:348] REPLICATION: starting
0112 02:58:11.312208 (+ 521us) log.cc:844] Serialized 432 byte log entry
0112 02:58:11.342709 (+ 30501us) op_driver.cc:464] REPLICATION: finished
0112 02:58:11.343153 (+ 444us) write_op.cc:301] APPLY: starting
0112 02:58:11.343222 (+ 69us) tablet.cc:1364] starting BulkCheckPresence
0112 02:58:11.343334 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0112 02:58:11.343436 (+ 102us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 02:58:11.343801 (+ 365us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 02:58:11.343949 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0112 02:58:11.343995 (+ 46us) write_op.cc:312] APPLY: finished
0112 02:58:11.344172 (+ 177us) log.cc:844] Serialized 28 byte log entry
0112 02:58:11.344338 (+ 166us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:58:11.344406 (+ 68us) write_op.cc:454] Released schema lock
0112 02:58:11.344450 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0112 02:58:11.344663 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0112 02:58:11.344690 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1086063,"Hive Metastore.run_cpu_time_us":650,"Hive Metastore.run_wall_time_us":212771,"child_traces":[["op",{"apply.queue_time_us":307,"apply.run_cpu_time_us":1472,"apply.run_wall_time_us":1472,"num_ops":2,"prepare.queue_time_us":515,"prepare.run_cpu_time_us":2680,"prepare.run_wall_time_us":7992,"raft.queue_time_us":130,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":15,"replication_time_us":30896}]]}
I20250112 02:58:11.357023 24520 tablet_bootstrap.cc:492] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.357467 24520 ts_tablet_manager.cc:1397] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.007s
I20250112 02:58:11.359521 23924 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_5c63a6d644b84d88901939668bf9dc24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:11.360417 23925 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41404:
table { table_name: "default.Table_5bafe92abae84e2798e614a3069d4084" } modify_external_catalogs: true
I20250112 02:58:11.360477 24520 raft_consensus.cc:357] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.361164 24520 raft_consensus.cc:383] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.361203 23925 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41404:
table { table_name: "default.Table_5bafe92abae84e2798e614a3069d4084" } modify_external_catalogs: true
I20250112 02:58:11.361445 24520 raft_consensus.cc:738] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.369927 24525 ts_tablet_manager.cc:1918] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.362043 24520 consensus_queue.cc:260] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.373083 24524 ts_tablet_manager.cc:1918] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.373104 24108 tablet_service.cc:1467] Processing CreateTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.373502 24524 log.cc:1198] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/d5a63ecb8eed451a8027d28bd2e8d91f
I20250112 02:58:11.370311 24525 log.cc:1198] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/d5a63ecb8eed451a8027d28bd2e8d91f
I20250112 02:58:11.374284 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.375725 24520 ts_tablet_manager.cc:1428] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 02:58:11.376576 24520 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:11.383214 24525 ts_tablet_manager.cc:1939] T d5a63ecb8eed451a8027d28bd2e8d91f P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:11.383178 24252 tablet_service.cc:1467] Processing CreateTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.384274 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.386453 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet d5a63ecb8eed451a8027d28bd2e8d91f (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:11.396136 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a63ecb8eed451a8027d28bd2e8d91f with delete_type TABLET_DATA_DELETED (replaced by tablet fba5ffea5d5f4bf1a734eccee2cfc3c8) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:11.396848 24526 tablet_replica.cc:331] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:11.397519 24526 raft_consensus.cc:2238] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.397980 24526 raft_consensus.cc:2267] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.401440 24526 ts_tablet_manager.cc:1905] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.382417 24524 ts_tablet_manager.cc:1939] T d5a63ecb8eed451a8027d28bd2e8d91f P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:11.406597 24521 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:11.407052 24521 ts_tablet_manager.cc:1397] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.001s
I20250112 02:58:11.409715 24521 raft_consensus.cc:357] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.410530 24521 raft_consensus.cc:383] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.410816 24521 raft_consensus.cc:738] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.416110 24521 consensus_queue.cc:260] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.426483 24521 ts_tablet_manager.cc:1428] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250112 02:58:11.427608 24520 tablet_bootstrap.cc:654] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.437551 24521 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:11.454586 24252 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3da24feed4401aaa584995fb696586 (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.455701 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3da24feed4401aaa584995fb696586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.457777 24107 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3da24feed4401aaa584995fb696586 (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.458856 24107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3da24feed4401aaa584995fb696586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.460100 24521 tablet_bootstrap.cc:654] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.475940 24520 tablet_bootstrap.cc:492] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.476357 24520 ts_tablet_manager.cc:1397] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.100s user 0.009s sys 0.003s
I20250112 02:58:11.478863 24520 raft_consensus.cc:357] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.479552 24520 raft_consensus.cc:383] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.479842 24520 raft_consensus.cc:738] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.481480 24520 consensus_queue.cc:260] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.483529 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.487530 24520 ts_tablet_manager.cc:1428] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250112 02:58:11.503017 23921 master_service.cc:946] ReplaceTablet: received request to replace tablet a3a91ada2bb547ceaf7b06a6a6764c74 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.510092 24521 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:11.510623 24521 ts_tablet_manager.cc:1397] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.006s
I20250112 02:58:11.513114 24521 raft_consensus.cc:357] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.513944 24521 raft_consensus.cc:383] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.514257 24521 raft_consensus.cc:738] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.515051 24521 consensus_queue.cc:260] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
F20250112 02:58:11.517665 23983 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_ac0f37d3057e4df2b361243d0f3e2103
*** Check failure stack trace: ***
I20250112 02:58:11.528541 23921 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet a3a91ada2bb547ceaf7b06a6a6764c74
I20250112 02:58:11.530647 24106 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 6363147706a947d4887a7395ee41c690) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:11.532426 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 6363147706a947d4887a7395ee41c690) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:11.532698 23921 catalog_manager.cc:6596] ReplaceTablet: tablet a3a91ada2bb547ceaf7b06a6a6764c74 deleted and replaced by tablet 6363147706a947d4887a7395ee41c690
I20250112 02:58:11.533098 24521 ts_tablet_manager.cc:1428] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250112 02:58:11.533222 24525 tablet_replica.cc:331] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:11.533933 24525 raft_consensus.cc:2238] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.534343 24525 raft_consensus.cc:2267] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.537645 24525 ts_tablet_manager.cc:1905] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.545962 24521 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:11.551250 24395 tablet_service.cc:1467] Processing CreateTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.552004 24521 tablet_bootstrap.cc:654] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.553404 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.557286 24393 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a91ada2bb547ceaf7b06a6a6764c74 with delete_type TABLET_DATA_DELETED (replaced by tablet 6363147706a947d4887a7395ee41c690) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:11.557219 24394 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3da24feed4401aaa584995fb696586 (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.558104 24524 tablet_replica.cc:331] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:11.558351 24394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b3da24feed4401aaa584995fb696586. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.558846 24524 raft_consensus.cc:2238] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.559437 24524 raft_consensus.cc:2267] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.561254 24531 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:11.561786 24531 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.563894 24524 ts_tablet_manager.cc:1905] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.562112 23921 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41404:
name: "default.Table_361969650c834c8baf1cd88e52f16210"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250112 02:58:11.567842 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:11.568284 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:11.569137 24530 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:11.568697 24531 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:11.590767 24525 ts_tablet_manager.cc:1918] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.600788 24524 ts_tablet_manager.cc:1918] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250112 02:58:11.595000 24206 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:11.603014 24207 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:11.606045 24207 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aff6bb94c7a14420a7540c9c198f0252; no voters: 048cd781735346e9a1e53c48b25f0e38, 4bea2270199d4f05ace161ff7f3fcfae
I20250112 02:58:11.607329 24531 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:11.608381 24525 log.cc:1198] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/a3a91ada2bb547ceaf7b06a6a6764c74
I20250112 02:58:11.609122 24525 ts_tablet_manager.cc:1939] T a3a91ada2bb547ceaf7b06a6a6764c74 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:11.609393 24526 ts_tablet_manager.cc:1918] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.609807 24526 log.cc:1198] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/d5a63ecb8eed451a8027d28bd2e8d91f
@ 0x7f172af0f950 google::LogMessage::Flush() at ??:0
I20250112 02:58:11.547417 23913 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet d5a63ecb8eed451a8027d28bd2e8d91f (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
W20250112 02:58:11.615051 23913 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.20.217.131:34845, user_credentials={real_user=slave}} blocked reactor thread for 67667.5us
I20250112 02:58:11.616196 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet a3a91ada2bb547ceaf7b06a6a6764c74 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
I20250112 02:58:11.616477 24524 log.cc:1198] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/a3a91ada2bb547ceaf7b06a6a6764c74
I20250112 02:58:11.617408 24524 ts_tablet_manager.cc:1939] T a3a91ada2bb547ceaf7b06a6a6764c74 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:11.619287 24520 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:11.625108 24520 tablet_bootstrap.cc:654] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.650821 24520 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.651247 24520 ts_tablet_manager.cc:1397] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.000s
I20250112 02:58:11.610415 24526 ts_tablet_manager.cc:1939] T d5a63ecb8eed451a8027d28bd2e8d91f P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:11.655844 24520 raft_consensus.cc:357] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.656549 24520 raft_consensus.cc:383] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.656666 24526 tablet_replica.cc:331] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:11.656845 24520 raft_consensus.cc:738] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.657047 23912 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet d5a63ecb8eed451a8027d28bd2e8d91f (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
@ 0x7f172af0f6bf google::LogMessage::~LogMessage() at ??:0
I20250112 02:58:11.659539 23913 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet a3a91ada2bb547ceaf7b06a6a6764c74 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
I20250112 02:58:11.661715 24521 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:11.662071 24521 ts_tablet_manager.cc:1397] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.004s
I20250112 02:58:11.664417 24521 raft_consensus.cc:357] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.665014 24521 raft_consensus.cc:383] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.665227 24521 raft_consensus.cc:738] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.668895 24252 tablet_service.cc:1467] Processing CreateTablet for tablet 6363147706a947d4887a7395ee41c690 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.669967 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6363147706a947d4887a7395ee41c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.669744 24393 tablet_service.cc:1467] Processing CreateTablet for tablet 6363147706a947d4887a7395ee41c690 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.670907 24393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6363147706a947d4887a7395ee41c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:11.681344 24519 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:11.657341 24526 raft_consensus.cc:2238] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:11.685878 24526 raft_consensus.cc:2267] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:11.687521 24519 tablet_bootstrap.cc:654] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.657490 24520 consensus_queue.cc:260] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
@ 0x7f1735e0a82a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250112 02:58:11.702049 24520 ts_tablet_manager.cc:1428] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20250112 02:58:11.703172 24520 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:11.716853 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.679332 24108 tablet_service.cc:1467] Processing CreateTablet for tablet 6363147706a947d4887a7395ee41c690 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:11.720115 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6363147706a947d4887a7395ee41c690. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f1735e0a07c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250112 02:58:11.722795 24521 consensus_queue.cc:260] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.723604 24531 raft_consensus.cc:491] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:11.723975 24520 tablet_bootstrap.cc:654] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.724028 24531 raft_consensus.cc:513] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.728327 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
W20250112 02:58:11.729817 24207 leader_election.cc:343] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:58:11.730901 24531 leader_election.cc:290] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845), 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:11.731475 24530 raft_consensus.cc:491] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:11.731520 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae" is_pre_election: true
I20250112 02:58:11.731930 24530 raft_consensus.cc:513] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.733609 24530 leader_election.cc:290] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:11.734393 24128 raft_consensus.cc:2463] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:11.735024 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252" is_pre_election: true
I20250112 02:58:11.735814 24272 raft_consensus.cc:2463] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 0.
I20250112 02:58:11.738619 24060 leader_election.cc:304] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:11.739568 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
W20250112 02:58:11.740753 24062 leader_election.cc:343] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:58:11.743111 24206 leader_election.cc:304] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters: 048cd781735346e9a1e53c48b25f0e38
I20250112 02:58:11.744144 24531 raft_consensus.cc:2798] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:11.744444 24531 raft_consensus.cc:491] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:11.744760 24531 raft_consensus.cc:3054] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:11.746120 24520 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.746474 24520 ts_tablet_manager.cc:1397] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.000s
I20250112 02:58:11.743353 24530 raft_consensus.cc:2798] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:11.746894 24530 raft_consensus.cc:491] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:11.747280 24530 raft_consensus.cc:3054] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:11.750945 24520 raft_consensus.cc:357] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.751600 24520 raft_consensus.cc:383] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.751874 24520 raft_consensus.cc:738] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.754814 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet a3a91ada2bb547ceaf7b06a6a6764c74 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.756222 24526 ts_tablet_manager.cc:1905] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:11.763761 24520 consensus_queue.cc:260] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.767385 24530 raft_consensus.cc:513] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.770097 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252"
I20250112 02:58:11.771167 24530 leader_election.cc:290] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Requested vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:11.774083 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
W20250112 02:58:11.781947 24062 leader_election.cc:343] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:58:11.786792 24519 tablet_bootstrap.cc:492] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:11.787204 24519 ts_tablet_manager.cc:1397] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.014s
@ 0x7f1735e0dd32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250112 02:58:11.789712 24519 raft_consensus.cc:357] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.766029 24531 raft_consensus.cc:513] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.791791 24272 raft_consensus.cc:2388] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bea2270199d4f05ace161ff7f3fcfae in current term 1: Already voted for candidate aff6bb94c7a14420a7540c9c198f0252 in this term.
I20250112 02:58:11.793586 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:11.795040 24531 leader_election.cc:290] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Requested vote from peers 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845), 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:11.795835 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3da24feed4401aaa584995fb696586" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae"
I20250112 02:58:11.790333 24519 raft_consensus.cc:383] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.797292 24519 raft_consensus.cc:738] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.798887 24060 leader_election.cc:304] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:11.798609 24519 consensus_queue.cc:260] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.799835 24415 raft_consensus.cc:3054] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:11.800568 24128 raft_consensus.cc:2388] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aff6bb94c7a14420a7540c9c198f0252 in current term 1: Already voted for candidate 4bea2270199d4f05ace161ff7f3fcfae in this term.
I20250112 02:58:11.801005 24530 raft_consensus.cc:2743] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250112 02:58:11.806943 24415 raft_consensus.cc:2463] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:11.810087 24207 leader_election.cc:304] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252; no voters: 4bea2270199d4f05ace161ff7f3fcfae
I20250112 02:58:11.810673 24521 ts_tablet_manager.cc:1428] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.148s user 0.009s sys 0.000s
I20250112 02:58:11.811753 24521 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:11.813828 24531 raft_consensus.cc:2798] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:11.814253 24531 raft_consensus.cc:695] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Becoming Leader. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Running, Role: LEADER
I20250112 02:58:11.815095 24531 consensus_queue.cc:237] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:11.828909 24521 tablet_bootstrap.cc:654] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.835060 24519 ts_tablet_manager.cc:1428] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.048s user 0.006s sys 0.001s
I20250112 02:58:11.851799 24520 ts_tablet_manager.cc:1428] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.105s user 0.017s sys 0.006s
I20250112 02:58:11.860265 24521 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:11.860685 24521 ts_tablet_manager.cc:1397] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.002s
I20250112 02:58:11.863328 24521 raft_consensus.cc:357] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.864171 24521 raft_consensus.cc:383] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.864495 24521 raft_consensus.cc:738] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.865111 24521 consensus_queue.cc:260] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.868680 24521 ts_tablet_manager.cc:1428] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
@ 0x7f1735e0dcea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250112 02:58:11.880883 24519 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:11.892009 23920 catalog_manager.cc:5526] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 reported cstate change: term changed from 0 to 1, leader changed from <none> to aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "aff6bb94c7a14420a7540c9c198f0252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } health_report { overall_health: UNKNOWN } } }
I20250112 02:58:11.902650 24520 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
@ 0x7f1735e0dc7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250112 02:58:11.907498 24519 tablet_bootstrap.cc:654] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.912590 24520 tablet_bootstrap.cc:654] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:11.932304 24519 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:11.932708 24519 ts_tablet_manager.cc:1397] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.004s
@ 0x7f1735e0dc42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250112 02:58:11.943447 24519 raft_consensus.cc:357] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.944159 24519 raft_consensus.cc:383] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.944417 24519 raft_consensus.cc:738] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.953750 23918 master_service.cc:946] ReplaceTablet: received request to replace tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.955286 24520 tablet_bootstrap.cc:492] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:11.955730 24520 ts_tablet_manager.cc:1397] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.000s
I20250112 02:58:11.957979 24520 raft_consensus.cc:357] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.958714 24520 raft_consensus.cc:383] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:11.959018 24520 raft_consensus.cc:738] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:11.959724 24520 consensus_queue.cc:260] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.961570 24520 ts_tablet_manager.cc:1428] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250112 02:58:11.974396 23919 master_service.cc:946] ReplaceTablet: received request to replace tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 from {username='slave'} at 127.0.0.1:41404
I20250112 02:58:11.953474 24519 consensus_queue.cc:260] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:11.979867 24519 ts_tablet_manager.cc:1428] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250112 02:58:11.980098 24526 ts_tablet_manager.cc:1918] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:11.980495 24526 log.cc:1198] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/a3a91ada2bb547ceaf7b06a6a6764c74
I20250112 02:58:11.981223 24526 ts_tablet_manager.cc:1939] T a3a91ada2bb547ceaf7b06a6a6764c74 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:11.982023 24519 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:11.984421 23912 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet a3a91ada2bb547ceaf7b06a6a6764c74 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
I20250112 02:58:12.002136 23918 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 2edd70f382cf4a44acb1db7b47eaf235: Sending DeleteTablet for 3 replicas of tablet fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.003834 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d870ffae55284213b10a7256f864a501) from {username='slave'} at 127.0.0.1:53670
I20250112 02:58:12.005028 24526 tablet_replica.cc:331] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:12.005717 24526 raft_consensus.cc:2238] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:12.006143 24526 raft_consensus.cc:2267] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:12.007445 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d870ffae55284213b10a7256f864a501) from {username='slave'} at 127.0.0.1:47790
I20250112 02:58:12.008226 24525 tablet_replica.cc:331] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:12.008383 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 with delete_type TABLET_DATA_DELETED (replaced by tablet d870ffae55284213b10a7256f864a501) from {username='slave'} at 127.0.0.1:40628
I20250112 02:58:12.009220 24525 raft_consensus.cc:2238] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:12.004402 23918 catalog_manager.cc:6596] ReplaceTablet: tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 deleted and replaced by tablet d870ffae55284213b10a7256f864a501
I20250112 02:58:12.009752 24525 raft_consensus.cc:2267] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250112 02:58:12.011932 23913 catalog_manager.cc:4852] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): delete failed for tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 already in progress: creating tablet
I20250112 02:58:12.012305 24525 ts_tablet_manager.cc:1905] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:12.018626 24526 ts_tablet_manager.cc:1905] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:12.024993 24519 tablet_bootstrap.cc:654] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:12.041049 24108 tablet_service.cc:1467] Processing CreateTablet for tablet d870ffae55284213b10a7256f864a501 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:12.041049 24395 tablet_service.cc:1467] Processing CreateTablet for tablet d870ffae55284213b10a7256f864a501 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:12.042151 24108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d870ffae55284213b10a7256f864a501. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:12.042151 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d870ffae55284213b10a7256f864a501. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:12.045506 24252 tablet_service.cc:1467] Processing CreateTablet for tablet d870ffae55284213b10a7256f864a501 (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:12.046579 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d870ffae55284213b10a7256f864a501. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f1735e0cf3e std::__1::__function::__func<>::operator()() at ??:0
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250112 02:58:12.082986 24519 tablet_bootstrap.cc:492] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:12.083401 24519 ts_tablet_manager.cc:1397] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.007s
I20250112 02:58:12.085541 24519 raft_consensus.cc:357] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.086166 24519 raft_consensus.cc:383] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:12.086396 24519 raft_consensus.cc:738] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:12.087294 24520 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:12.089680 24521 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:12.092581 24520 tablet_bootstrap.cc:654] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:12.094841 24521 tablet_bootstrap.cc:654] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:12.095676 24519 consensus_queue.cc:260] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
W20250112 02:58:12.104339 23918 catalog_manager.cc:5359] Ignoring report from unknown tablet fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.108276 24526 ts_tablet_manager.cc:1918] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:12.108649 24526 log.cc:1198] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.108868 24521 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:12.109274 24521 ts_tablet_manager.cc:1397] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20250112 02:58:12.110450 24519 ts_tablet_manager.cc:1428] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20250112 02:58:12.111748 24521 raft_consensus.cc:357] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.112385 24521 raft_consensus.cc:383] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:12.112643 24521 raft_consensus.cc:738] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:12.113301 24521 consensus_queue.cc:260] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
W20250112 02:58:12.120947 23920 catalog_manager.cc:5359] Ignoring report from unknown tablet fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.124378 24519 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:12.127754 24521 ts_tablet_manager.cc:1428] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250112 02:58:12.131865 24520 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:12.132246 24520 ts_tablet_manager.cc:1397] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.000s
I20250112 02:58:12.134270 24526 ts_tablet_manager.cc:1939] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:12.113302 24525 ts_tablet_manager.cc:1918] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:12.137298 24525 log.cc:1198] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.138010 24520 raft_consensus.cc:357] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.139201 24520 raft_consensus.cc:383] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:12.139438 24520 raft_consensus.cc:738] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:12.140118 23912 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:12.141402 24525 ts_tablet_manager.cc:1939] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:12.141237 24520 consensus_queue.cc:260] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.145915 23911 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet fba5ffea5d5f4bf1a734eccee2cfc3c8 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:12.149245 24519 tablet_bootstrap.cc:654] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:12.157617 24520 ts_tablet_manager.cc:1428] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
@ 0x7f172d1548d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
I20250112 02:58:12.174639 24519 tablet_bootstrap.cc:492] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:12.175136 24519 ts_tablet_manager.cc:1397] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.002s
I20250112 02:58:12.178299 24519 raft_consensus.cc:357] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.179268 24519 raft_consensus.cc:383] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:12.179558 24519 raft_consensus.cc:738] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:12.180258 24519 consensus_queue.cc:260] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.182828 24519 ts_tablet_manager.cc:1428] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250112 02:58:12.191556 23920 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
@ 0x7f172e7d36db start_thread at ??:0
I20250112 02:58:12.216885 23918 master_service.cc:946] ReplaceTablet: received request to replace tablet b6c6df7f45f04d46b788c13f1c3f2455 from {username='slave'} at 127.0.0.1:41404
@ 0x7f1729bb471f clone at ??:0
I20250112 02:58:12.305474 24574 raft_consensus.cc:491] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:12.305959 24574 raft_consensus.cc:513] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.307998 24574 leader_election.cc:290] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:12.309304 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d870ffae55284213b10a7256f864a501" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
I20250112 02:58:12.309823 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d870ffae55284213b10a7256f864a501" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252" is_pre_election: true
I20250112 02:58:12.310088 24415 raft_consensus.cc:2463] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 0.
I20250112 02:58:12.310740 24272 raft_consensus.cc:2463] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 0.
I20250112 02:58:12.311482 24062 leader_election.cc:304] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, 4bea2270199d4f05ace161ff7f3fcfae; no voters:
I20250112 02:58:12.312404 24574 raft_consensus.cc:2798] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:12.312685 24574 raft_consensus.cc:491] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:12.312963 24574 raft_consensus.cc:3054] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.319219 24574 raft_consensus.cc:513] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.321010 24574 leader_election.cc:290] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Requested vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:12.322273 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d870ffae55284213b10a7256f864a501" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:12.322491 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d870ffae55284213b10a7256f864a501" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252"
I20250112 02:58:12.322971 24415 raft_consensus.cc:3054] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.323431 24272 raft_consensus.cc:3054] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.332512 24415 raft_consensus.cc:2463] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 1.
I20250112 02:58:12.333884 24062 leader_election.cc:304] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, 4bea2270199d4f05ace161ff7f3fcfae; no voters:
I20250112 02:58:12.339140 24272 raft_consensus.cc:2463] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 1.
I20250112 02:58:12.343060 24574 raft_consensus.cc:2798] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:12.353547 24574 raft_consensus.cc:695] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 LEADER]: Becoming Leader. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Running, Role: LEADER
I20250112 02:58:12.355837 24574 consensus_queue.cc:237] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.406052 24531 raft_consensus.cc:491] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:12.406767 24531 raft_consensus.cc:513] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.410764 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6363147706a947d4887a7395ee41c690" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae" is_pre_election: true
I20250112 02:58:12.411636 24128 raft_consensus.cc:2463] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:12.413384 24206 leader_election.cc:304] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:12.415663 24531 leader_election.cc:290] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:12.416415 24531 raft_consensus.cc:2798] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:12.416501 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6363147706a947d4887a7395ee41c690" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
I20250112 02:58:12.416945 24531 raft_consensus.cc:491] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:12.417382 24531 raft_consensus.cc:3054] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.417328 24415 raft_consensus.cc:2463] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:12.442394 24531 raft_consensus.cc:513] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.450116 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6363147706a947d4887a7395ee41c690" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae"
I20250112 02:58:12.450986 24128 raft_consensus.cc:3054] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.454360 24415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6363147706a947d4887a7395ee41c690" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:12.455139 24415 raft_consensus.cc:3054] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:12.461632 24128 raft_consensus.cc:2463] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:12.462204 24531 leader_election.cc:290] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Requested vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:12.463830 24206 leader_election.cc:304] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:12.465294 24531 raft_consensus.cc:2798] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:12.466851 24531 raft_consensus.cc:695] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Becoming Leader. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Running, Role: LEADER
I20250112 02:58:12.468067 24531 consensus_queue.cc:237] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.498919 24415 raft_consensus.cc:2463] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:12.585310 24531 consensus_queue.cc:1035] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:12.651614 24599 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2edd70f382cf4a44acb1db7b47eaf235)
I20250112 02:58:12.652098 24599 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:12.670346 24531 consensus_queue.cc:1035] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:58:12.676064 24599 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:12.683655 23835 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
W20250112 02:58:12.779595 24351 connection.cc:537] client connection to 127.20.217.188:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:12.779937 24474 connection.cc:537] client connection to 127.20.217.188:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:12.782320 24474 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.217.188:34169)
W20250112 02:58:12.782763 24208 connection.cc:537] client connection to 127.20.217.188:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:12.782786 24063 connection.cc:537] client connection to 127.20.217.188:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:12.785178 23836 connection.cc:537] client connection to 127.20.217.188:34169 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:12.786887 24539 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:58:12.781275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.217.188:34169 (local address 127.20.217.131:36557)
0112 02:58:12.782121 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0112 02:58:12.782160 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0112 02:58:12.782384 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0112 02:58:12.785770 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":284}
W20250112 02:58:12.789474 24474 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.20.217.188:34169)
W20250112 02:58:12.790681 24474 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.20.217.188:34169)
W20250112 02:58:12.793947 24474 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.20.217.188:34169)
W20250112 02:58:12.795055 23836 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:12.795435 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:12.796113 24599 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250112 02:58:12.817773 24529 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:12.818300 24529 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:12.820245 24529 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:12.824509 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:12.825724 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:12.826148 24349 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:12.827399 24529 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:13.158413 24574 consensus_queue.cc:1035] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:13.236909 24574 consensus_queue.cc:1035] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:13.330963 24621 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2edd70f382cf4a44acb1db7b47eaf235)
I20250112 02:58:13.331535 24621 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:13.333735 24621 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:11.152972 24554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:11.145642 24549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:13.351594 23835 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757): Network error: Client connection negotiation failed: client connection to 127.20.217.190:33757: connect: Connection refused (error 111)
W20250112 02:58:13.353447 23836 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:13.353924 23836 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 98781ea331e8470ba27c6fc976a70be7
I20250112 02:58:13.354660 24621 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250112 02:58:13.355614 24551 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2209 milliseconds
I20250112 02:58:13.355743 24510 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250112 02:58:13.356984 24510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:13.359617 24510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:13.360944 24510 hybrid_clock.cc:648] HybridClock initialized: now 1736650693360909 us; error 30 us; skew 500 ppm
I20250112 02:58:13.361713 24510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:13.373265 24510 webserver.cc:458] Webserver started at http://127.20.217.190:46241/ using document root <none> and password file <none>
I20250112 02:58:13.374145 24510 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:13.374353 24510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:13.382997 24510 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250112 02:58:13.387373 24629 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:13.388388 24510 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250112 02:58:13.388717 24510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
uuid: "98781ea331e8470ba27c6fc976a70be7"
format_stamp: "Formatted at 2025-01-12 02:57:58 on dist-test-slave-gppw"
I20250112 02:58:13.391168 24510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:13.432991 24616 consensus_queue.cc:1035] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:58:13.453354 24616 consensus_queue.cc:1035] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:58:13.456342 24510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:13.458593 24510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:13.459319 24510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:13.569631 24510 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.190:33757
I20250112 02:58:13.569766 24683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.190:33757 every 8 connection(s)
I20250112 02:58:13.572458 24510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
I20250112 02:58:13.578238 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 24510
I20250112 02:58:13.584220 24684 sys_catalog.cc:263] Verifying existing consensus state
I20250112 02:58:13.597483 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap starting.
I20250112 02:58:13.621454 24319 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:13.619657 24684 log.cc:826] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:13.643932 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:58:13.644856 24684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap complete.
I20250112 02:58:13.672477 24684 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:13.675750 24684 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98781ea331e8470ba27c6fc976a70be7, State: Initialized, Role: FOLLOWER
I20250112 02:58:13.676576 24684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:13.679426 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:13.680001 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:13.681304 24684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:58:13.742370 24704 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:13.742689 24704 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: success
I20250112 02:58:13.750645 24704 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: acquiring CA information for follower catalog manager: success
I20250112 02:58:13.755976 24704 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:58:13.752380 24510 master_runner.cc:186] Error getting master registration for 127.20.217.188:34169: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111),
I20250112 02:58:13.773226 24510 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:13.980751 24176 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:13.985121 24463 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:13.985706 24463 heartbeater.cc:502] Master 127.20.217.190:33757 was elected leader, sending a full tablet report...
I20250112 02:58:13.990984 24649 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" instance_seqno: 1736650685450797) as {username='slave'} at 127.20.217.129:37711; Asking this server to re-register.
I20250112 02:58:13.992148 24648 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" instance_seqno: 1736650689444137) as {username='slave'} at 127.20.217.131:39767; Asking this server to re-register.
I20250112 02:58:13.993140 24176 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:13.993762 24463 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:13.996573 24648 ts_manager.cc:194] Registered new tserver with Master: 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:13.996933 24649 ts_manager.cc:194] Registered new tserver with Master: 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:14.092201 24690 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:14.092733 24690 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:14.095671 24690 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:14.097234 23857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98781ea331e8470ba27c6fc976a70be7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ab49e495ba34dc29d914b277a872014" is_pre_election: true
I20250112 02:58:14.098018 23857 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98781ea331e8470ba27c6fc976a70be7 in current term 2: Already voted for candidate 2edd70f382cf4a44acb1db7b47eaf235 in this term.
W20250112 02:58:14.099464 24633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 02:58:14.103092 24633 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:14.103479 24633 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:14.104168 24690 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 1 FOLLOWER]: Advancing to term 2
I20250112 02:58:14.110221 24690 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 02:58:14.133185 24649 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" instance_seqno: 1736650687559117) as {username='slave'} at 127.20.217.130:35009; Asking this server to re-register.
I20250112 02:58:14.134850 24319 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:14.138265 24649 ts_manager.cc:194] Registered new tserver with Master: aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
I20250112 02:58:14.479537 24715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2edd70f382cf4a44acb1db7b47eaf235)
I20250112 02:58:14.480012 24715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:14.488950 24715 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:14.495110 23836 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:14.508255 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7" is_pre_election: true
I20250112 02:58:14.509084 24659 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab49e495ba34dc29d914b277a872014 in term 2.
I20250112 02:58:14.510200 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235
I20250112 02:58:14.511241 24715 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250112 02:58:14.511550 24715 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2edd70f382cf4a44acb1db7b47eaf235)
I20250112 02:58:14.511854 24715 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:58:14.519176 24715 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:14.521638 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6ab49e495ba34dc29d914b277a872014" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: false dest_uuid: "98781ea331e8470ba27c6fc976a70be7"
I20250112 02:58:14.522193 24659 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 2 FOLLOWER]: Advancing to term 3
I20250112 02:58:14.528482 24659 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab49e495ba34dc29d914b277a872014 in term 3.
I20250112 02:58:14.529449 23835 leader_election.cc:304] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ab49e495ba34dc29d914b277a872014, 98781ea331e8470ba27c6fc976a70be7; no voters:
I20250112 02:58:14.533890 24715 leader_election.cc:290] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 election: Requested vote from peers 98781ea331e8470ba27c6fc976a70be7 (127.20.217.190:33757), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
I20250112 02:58:14.534371 24715 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 3 FOLLOWER]: Leader election won for term 3
I20250112 02:58:14.534796 24715 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [term 3 LEADER]: Becoming Leader. State: Replica: 6ab49e495ba34dc29d914b277a872014, State: Running, Role: LEADER
I20250112 02:58:14.535662 24715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
W20250112 02:58:14.539780 23836 leader_election.cc:336] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:14.545228 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab49e495ba34dc29d914b277a872014. Latest consensus state: current_term: 3 leader_uuid: "6ab49e495ba34dc29d914b277a872014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:14.545964 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: LEADER
I20250112 02:58:14.547647 24722 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 02:58:14.558467 24722 catalog_manager.cc:670] Loaded metadata for table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]
I20250112 02:58:14.560165 24722 catalog_manager.cc:670] Loaded metadata for table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]
I20250112 02:58:14.561677 24722 catalog_manager.cc:670] Loaded metadata for table default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]
I20250112 02:58:14.568713 24722 tablet_loader.cc:96] loaded metadata for tablet 5b3da24feed4401aaa584995fb696586 (table default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a])
I20250112 02:58:14.569967 24722 tablet_loader.cc:96] loaded metadata for tablet 6363147706a947d4887a7395ee41c690 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a])
I20250112 02:58:14.571137 24722 tablet_loader.cc:96] loaded metadata for tablet d870ffae55284213b10a7256f864a501 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e])
I20250112 02:58:14.572474 24722 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 02:58:14.577102 24722 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:14.577358 24722 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 02:58:14.582664 24722 catalog_manager.cc:1505] Loading token signing keys...
I20250112 02:58:14.587121 24722 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Loaded TSK: 0
I20250112 02:58:14.588362 24722 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 02:58:14.592955 24722 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250112 02:58:14.597407 24722 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250112 02:58:14.639662 23848 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b3da24feed4401aaa584995fb696586 from {username='slave'} at 127.0.0.1:58426
I20250112 02:58:14.641047 23846 master_service.cc:946] ReplaceTablet: received request to replace tablet 6363147706a947d4887a7395ee41c690 from {username='slave'} at 127.0.0.1:58426
I20250112 02:58:14.647325 23849 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58426:
name: "default.Table_361969650c834c8baf1cd88e52f16210"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:14.648833 23843 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58426:
name: "default.Table_ebcfbc2fbb6442918a4444cc0bcbab4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:14.650282 23842 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58426:
name: "default.Table_76f3bea64f5b4d869d3c8ec3ef52189e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 02:58:14.638909 23847 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58426:
table { table_name: "default.Table_5bafe92abae84e2798e614a3069d4084" } modify_external_catalogs: true
I20250112 02:58:14.649278 23844 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58426:
table { table_name: "default.Table_9ceac5e9837d4fe99f40d0fd14a67b2b" } new_table_name: "default.Table_f1941917eac84909bd5258d5518637b6" modify_external_catalogs: true
I20250112 02:58:14.650436 23841 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:58426:
table { table_name: "default.Table_51ffd08ccf99400ba427786f23d0bcfd" } new_table_name: "default.Table_a9f5d2a7adb243439a6be2d0b74d48c0" modify_external_catalogs: true
I20250112 02:58:14.667981 23847 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58426:
table { table_name: "default.Table_5bafe92abae84e2798e614a3069d4084" } modify_external_catalogs: true
I20250112 02:58:14.646400 23845 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58426:
name: "default.Table_5c63a6d644b84d88901939668bf9dc24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20250112 02:58:14.674429 24035 hms_notification_log_listener.cc:263] Received out-of-order notification log event (last processed event ID: 1): 1 CREATE_TABLE default.table_ac0f37d3057e4df2b361243d0f3e2103
*** Check failure stack trace: ***
I20250112 02:58:14.677913 24659 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Refusing update from remote peer 6ab49e495ba34dc29d914b277a872014: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250112 02:58:14.680018 24715 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 4, Time since last communication: 0.000s
@ 0x7fa218226950 google::LogMessage::Flush() at ??:0
W20250112 02:58:14.705128 23836 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 -> Peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Couldn't send request to peer 2edd70f382cf4a44acb1db7b47eaf235. Status: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fa2182266bf google::LogMessage::~LogMessage() at ??:0
I20250112 02:58:14.737901 24724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ab49e495ba34dc29d914b277a872014. Latest consensus state: current_term: 3 leader_uuid: "6ab49e495ba34dc29d914b277a872014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:14.738853 24724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:14.738935 24723 mvcc.cc:204] Tried to move back new op lower bound from 7113321245381734400 to 7113321244827000832. Current Snapshot: MvccSnapshot[applied={T|T < 7113321245381734400}]
I20250112 02:58:14.735992 24728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6ab49e495ba34dc29d914b277a872014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:14.740371 24728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fa22312182a kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20250112 02:58:14.743093 24728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6ab49e495ba34dc29d914b277a872014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:14.743784 24728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:14.760447 23846 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Sending DeleteTablet for 3 replicas of tablet 6363147706a947d4887a7395ee41c690
@ 0x7fa22312107c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20250112 02:58:14.774678 23846 catalog_manager.cc:6596] ReplaceTablet: tablet 6363147706a947d4887a7395ee41c690 deleted and replaced by tablet 112bccd320884506853e7b3870ca8440
I20250112 02:58:14.811333 24715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6ab49e495ba34dc29d914b277a872014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:14.812069 24715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014 [sys.catalog]: This master's current role is: LEADER
I20250112 02:58:14.816149 24252 tablet_service.cc:1514] Processing DeleteTablet for tablet 6363147706a947d4887a7395ee41c690 with delete_type TABLET_DATA_DELETED (replaced by tablet 112bccd320884506853e7b3870ca8440) from {username='slave'} at 127.0.0.1:47806
I20250112 02:58:14.816776 24395 tablet_service.cc:1514] Processing DeleteTablet for tablet 6363147706a947d4887a7395ee41c690 with delete_type TABLET_DATA_DELETED (replaced by tablet 112bccd320884506853e7b3870ca8440) from {username='slave'} at 127.0.0.1:40634
I20250112 02:58:14.817318 23848 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Sending DeleteTablet for 3 replicas of tablet 5b3da24feed4401aaa584995fb696586
I20250112 02:58:14.818795 23848 catalog_manager.cc:6596] ReplaceTablet: tablet 5b3da24feed4401aaa584995fb696586 deleted and replaced by tablet e81458757d454d059ba307c6d228be9a
I20250112 02:58:14.820268 24393 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3da24feed4401aaa584995fb696586 with delete_type TABLET_DATA_DELETED (replaced by tablet e81458757d454d059ba307c6d228be9a) from {username='slave'} at 127.0.0.1:40634
I20250112 02:58:14.821552 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3da24feed4401aaa584995fb696586 with delete_type TABLET_DATA_DELETED (replaced by tablet e81458757d454d059ba307c6d228be9a) from {username='slave'} at 127.0.0.1:47806
@ 0x7fa223124d32 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20250112 02:58:14.835525 24741 tablet_replica.cc:331] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:14.836289 24741 raft_consensus.cc:2238] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:58:14.836133 24251 tablet_service.cc:1467] Processing CreateTablet for tablet 112bccd320884506853e7b3870ca8440 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.836516 24742 tablet_replica.cc:331] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:14.837270 24251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112bccd320884506853e7b3870ca8440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.837286 24742 raft_consensus.cc:2238] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:14.837231 24741 raft_consensus.cc:2267] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.837779 24742 raft_consensus.cc:2267] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.839190 24741 ts_tablet_manager.cc:1905] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:14.839716 24742 ts_tablet_manager.cc:1905] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:14.848156 24395 tablet_service.cc:1467] Processing CreateTablet for tablet 112bccd320884506853e7b3870ca8440 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.849224 24395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112bccd320884506853e7b3870ca8440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.852821 24741 ts_tablet_manager.cc:1918] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:14.853138 24741 log.cc:1198] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/6363147706a947d4887a7395ee41c690
I20250112 02:58:14.853821 24741 ts_tablet_manager.cc:1939] T 6363147706a947d4887a7395ee41c690 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:14.856484 23833 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet 6363147706a947d4887a7395ee41c690 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
I20250112 02:58:14.857434 24741 tablet_replica.cc:331] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:14.859369 24741 raft_consensus.cc:2238] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:58:14.860144 24741 raft_consensus.cc:2267] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.861817 24741 ts_tablet_manager.cc:1905] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:14.867602 24252 tablet_service.cc:1467] Processing CreateTablet for tablet e81458757d454d059ba307c6d228be9a (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.868570 24252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81458757d454d059ba307c6d228be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.870234 24742 ts_tablet_manager.cc:1918] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:14.870604 24742 log.cc:1198] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/6363147706a947d4887a7395ee41c690
I20250112 02:58:14.871357 24742 ts_tablet_manager.cc:1939] T 6363147706a947d4887a7395ee41c690 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:14.871279 24393 tablet_service.cc:1467] Processing CreateTablet for tablet e81458757d454d059ba307c6d228be9a (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.872316 24393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81458757d454d059ba307c6d228be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.880789 24742 tablet_replica.cc:331] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:14.882050 23835 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet 6363147706a947d4887a7395ee41c690 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
@ 0x7fa223124cea _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20250112 02:58:14.885998 24108 tablet_service.cc:1514] Processing DeleteTablet for tablet 6363147706a947d4887a7395ee41c690 with delete_type TABLET_DATA_DELETED (replaced by tablet 112bccd320884506853e7b3870ca8440) from {username='slave'} at 127.0.0.1:53686
I20250112 02:58:14.887157 24107 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3da24feed4401aaa584995fb696586 with delete_type TABLET_DATA_DELETED (replaced by tablet e81458757d454d059ba307c6d228be9a) from {username='slave'} at 127.0.0.1:53686
I20250112 02:58:14.887527 24106 tablet_service.cc:1467] Processing CreateTablet for tablet 112bccd320884506853e7b3870ca8440 (DEFAULT_TABLE table=default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.888965 24106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 112bccd320884506853e7b3870ca8440. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.891475 24105 tablet_service.cc:1467] Processing CreateTablet for tablet e81458757d454d059ba307c6d228be9a (DEFAULT_TABLE table=default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:14.892508 24105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e81458757d454d059ba307c6d228be9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:14.904187 24746 tablet_replica.cc:331] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:14.905442 24746 raft_consensus.cc:2238] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:14.905725 24747 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:14.908290 24741 ts_tablet_manager.cc:1918] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:14.908661 24741 log.cc:1198] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/5b3da24feed4401aaa584995fb696586
I20250112 02:58:14.909447 24741 ts_tablet_manager.cc:1939] T 5b3da24feed4401aaa584995fb696586 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:14.910636 24746 raft_consensus.cc:2267] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.912590 24746 ts_tablet_manager.cc:1905] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:14.912655 23833 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet 5b3da24feed4401aaa584995fb696586 (table default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]) successfully deleted
I20250112 02:58:14.914682 24742 raft_consensus.cc:2238] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:14.915180 24742 raft_consensus.cc:2267] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.916325 24745 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:14.920760 24725 mvcc.cc:204] Tried to move back new op lower bound from 7113321245402648576 to 7113321225807716352. Current Snapshot: MvccSnapshot[applied={T|T < 7113321245402648576 or (T in {7113321245402648576})}]
I20250112 02:58:14.921814 24742 ts_tablet_manager.cc:1905] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:14.926263 24745 tablet_bootstrap.cc:654] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:14.929109 24747 tablet_bootstrap.cc:654] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:14.930151 24746 ts_tablet_manager.cc:1918] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:14.930522 24746 log.cc:1198] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/6363147706a947d4887a7395ee41c690
I20250112 02:58:14.937484 24747 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:14.937928 24747 ts_tablet_manager.cc:1397] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20250112 02:58:14.940373 24747 raft_consensus.cc:357] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:14.941044 24747 raft_consensus.cc:383] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:14.941313 24747 raft_consensus.cc:738] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:14.942055 24747 consensus_queue.cc:260] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:14.949451 24747 ts_tablet_manager.cc:1428] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250112 02:58:14.951364 24747 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:14.953658 24318 heartbeater.cc:502] Master 127.20.217.189:40327 was elected leader, sending a full tablet report...
I20250112 02:58:14.931324 24746 ts_tablet_manager.cc:1939] T 6363147706a947d4887a7395ee41c690 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:14.959794 24747 tablet_bootstrap.cc:654] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:14.961486 24746 tablet_replica.cc:331] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:14.962225 24746 raft_consensus.cc:2238] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:14.962752 24746 raft_consensus.cc:2267] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:14.962648 23834 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet 6363147706a947d4887a7395ee41c690 (table default.table_9ceac5e9837d4fe99f40d0fd14a67b2b [id=0f37e561467548519e44e122c0fe2d0a]) successfully deleted
I20250112 02:58:14.952284 24751 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:14.961012 23846 catalog_manager.cc:5526] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129). New cstate: current_term: 1 leader_uuid: "4bea2270199d4f05ace161ff7f3fcfae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } }
I20250112 02:58:14.972229 24745 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:14.972839 24745 ts_tablet_manager.cc:1397] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.007s
I20250112 02:58:14.978555 24745 raft_consensus.cc:357] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:14.979274 24745 raft_consensus.cc:383] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:14.979537 24745 raft_consensus.cc:738] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:14.980144 24745 consensus_queue.cc:260] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:14.985116 23848 master_service.cc:946] ReplaceTablet: received request to replace tablet 5b3da24feed4401aaa584995fb696586 from {username='slave'} at 127.0.0.1:58426
I20250112 02:58:14.986186 24747 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:14.986896 24747 ts_tablet_manager.cc:1397] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.000s
I20250112 02:58:14.990237 24747 raft_consensus.cc:357] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:14.990837 24616 raft_consensus.cc:491] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:14.991092 24747 raft_consensus.cc:383] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:14.991449 24747 raft_consensus.cc:738] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:14.991338 24616 raft_consensus.cc:513] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:14.992210 24747 consensus_queue.cc:260] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:14.993156 24616 leader_election.cc:290] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845), 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:14.994925 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112bccd320884506853e7b3870ca8440" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
I20250112 02:58:14.995489 24414 raft_consensus.cc:2463] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:14.997597 24207 leader_election.cc:304] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:14.994211 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112bccd320884506853e7b3870ca8440" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae" is_pre_election: true
W20250112 02:58:14.999301 24206 leader_election.cc:343] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:58:14.999317 24616 raft_consensus.cc:2798] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:15.000268 24616 raft_consensus.cc:491] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:15.000571 24616 raft_consensus.cc:3054] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.001550 24751 tablet_bootstrap.cc:654] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:15.002117 24747 ts_tablet_manager.cc:1428] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20250112 02:58:15.008467 24616 raft_consensus.cc:513] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:15.015456 24462 heartbeater.cc:502] Master 127.20.217.189:40327 was elected leader, sending a full tablet report...
I20250112 02:58:15.015697 24742 ts_tablet_manager.cc:1918] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.016075 24742 log.cc:1198] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/5b3da24feed4401aaa584995fb696586
I20250112 02:58:15.016805 24742 ts_tablet_manager.cc:1939] T 5b3da24feed4401aaa584995fb696586 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:15.016813 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112bccd320884506853e7b3870ca8440" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:15.017439 24414 raft_consensus.cc:3054] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.019857 24745 ts_tablet_manager.cc:1428] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20250112 02:58:15.021924 24745 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
W20250112 02:58:15.024246 23848 catalog_manager.cc:5359] Ignoring report from unknown tablet fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:15.027614 24745 tablet_bootstrap.cc:654] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:15.027812 23840 master_service.cc:946] ReplaceTablet: received request to replace tablet d870ffae55284213b10a7256f864a501 from {username='slave'} at 127.0.0.1:58426
@ 0x7fa223124c7a std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20250112 02:58:15.033100 23835 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet 5b3da24feed4401aaa584995fb696586 (table default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]) successfully deleted
I20250112 02:58:15.035970 24414 raft_consensus.cc:2463] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:15.043654 24616 leader_election.cc:290] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Requested vote from peers 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845), 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:15.044303 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "112bccd320884506853e7b3870ca8440" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae"
I20250112 02:58:15.044638 24207 leader_election.cc:304] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252; no voters:
W20250112 02:58:15.045349 24206 leader_election.cc:343] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Illegal state: must be running to vote when last-logged opid is not known
I20250112 02:58:15.047362 24616 raft_consensus.cc:2798] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:15.048974 24616 raft_consensus.cc:695] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Becoming Leader. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Running, Role: LEADER
I20250112 02:58:15.049819 24616 consensus_queue.cc:237] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:15.051865 23840 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Sending DeleteTablet for 3 replicas of tablet d870ffae55284213b10a7256f864a501
I20250112 02:58:15.053333 23840 catalog_manager.cc:6596] ReplaceTablet: tablet d870ffae55284213b10a7256f864a501 deleted and replaced by tablet 4fbc0b4256c94a9094c2b8f402056b2c
I20250112 02:58:15.055574 24746 ts_tablet_manager.cc:1905] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.057723 24745 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:15.058153 24745 ts_tablet_manager.cc:1397] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.001s
I20250112 02:58:15.060756 24745 raft_consensus.cc:357] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.061234 24393 tablet_service.cc:1514] Processing DeleteTablet for tablet d870ffae55284213b10a7256f864a501 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fbc0b4256c94a9094c2b8f402056b2c) from {username='slave'} at 127.0.0.1:40634
I20250112 02:58:15.061301 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet d870ffae55284213b10a7256f864a501 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fbc0b4256c94a9094c2b8f402056b2c) from {username='slave'} at 127.0.0.1:53686
I20250112 02:58:15.061902 24742 tablet_replica.cc:331] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:15.062611 24742 raft_consensus.cc:2238] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:15.063069 24742 raft_consensus.cc:2267] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.064847 24742 ts_tablet_manager.cc:1905] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.065896 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet d870ffae55284213b10a7256f864a501 with delete_type TABLET_DATA_DELETED (replaced by tablet 4fbc0b4256c94a9094c2b8f402056b2c) from {username='slave'} at 127.0.0.1:47806
I20250112 02:58:15.067427 24741 tablet_replica.cc:331] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:15.068162 24741 raft_consensus.cc:2238] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:15.068715 24741 raft_consensus.cc:2267] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.070520 24741 ts_tablet_manager.cc:1905] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.072036 24393 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3da24feed4401aaa584995fb696586 with delete_type TABLET_DATA_DELETED (replaced by tablet e81458757d454d059ba307c6d228be9a) from {username='slave'} at 127.0.0.1:40634
@ 0x7fa223124c42 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20250112 02:58:15.061448 24745 raft_consensus.cc:383] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.094681 24745 raft_consensus.cc:738] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
I20250112 02:58:15.095292 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc0b4256c94a9094c2b8f402056b2c (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:15.095446 24745 consensus_queue.cc:260] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.096324 24105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbc0b4256c94a9094c2b8f402056b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:15.101239 24251 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc0b4256c94a9094c2b8f402056b2c (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:15.102265 24251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbc0b4256c94a9094c2b8f402056b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:15.103444 24745 ts_tablet_manager.cc:1428] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.045s user 0.007s sys 0.001s
I20250112 02:58:15.107344 24393 tablet_service.cc:1467] Processing CreateTablet for tablet 4fbc0b4256c94a9094c2b8f402056b2c (DEFAULT_TABLE table=default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 02:58:15.108357 24393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fbc0b4256c94a9094c2b8f402056b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 02:58:15.120827 23848 catalog_manager.cc:5526] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 reported cstate change: term changed from 0 to 1, leader changed from <none> to aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130). New cstate: current_term: 1 leader_uuid: "aff6bb94c7a14420a7540c9c198f0252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } health_report { overall_health: UNKNOWN } } }
I20250112 02:58:15.126565 24741 ts_tablet_manager.cc:1918] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.127020 24741 log.cc:1198] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/d870ffae55284213b10a7256f864a501
I20250112 02:58:15.127802 24741 ts_tablet_manager.cc:1939] T d870ffae55284213b10a7256f864a501 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:15.130580 23833 catalog_manager.cc:4872] TS aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): tablet d870ffae55284213b10a7256f864a501 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
I20250112 02:58:15.134208 24742 ts_tablet_manager.cc:1918] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.147316 24742 log.cc:1198] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/d870ffae55284213b10a7256f864a501
I20250112 02:58:15.147490 24751 tablet_bootstrap.cc:492] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:15.147941 24751 ts_tablet_manager.cc:1397] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.196s user 0.006s sys 0.005s
I20250112 02:58:15.148262 24742 ts_tablet_manager.cc:1939] T d870ffae55284213b10a7256f864a501 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:15.151158 23835 catalog_manager.cc:4872] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): tablet d870ffae55284213b10a7256f864a501 (table default.table_51ffd08ccf99400ba427786f23d0bcfd [id=814d533e9fb741cab793d3e6a98ee15e]) successfully deleted
@ 0x7fa223123f3e std::__1::__function::__func<>::operator()() at ??:0
W20250112 02:58:15.152088 23835 catalog_manager.cc:4839] TS 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): delete failed for tablet 5b3da24feed4401aaa584995fb696586 because the tablet was not found. No further retry: Not found: Tablet not found: 5b3da24feed4401aaa584995fb696586
@ 0x59ea05 std::__1::__function::__value_func<>::operator()() at ??:?
I20250112 02:58:15.154078 24751 raft_consensus.cc:357] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.155017 24751 raft_consensus.cc:383] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.155340 24751 raft_consensus.cc:738] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
@ 0x59e83a std::__1::function<>::operator()() at ??:?
I20250112 02:58:15.157027 24747 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252: Bootstrap starting.
I20250112 02:58:15.158257 24745 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38: Bootstrap starting.
I20250112 02:58:15.163463 24747 tablet_bootstrap.cc:654] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252: Neither blocks nor log segments found. Creating new log.
W20250112 02:58:15.165387 24062 consensus_peers.cc:487] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae -> Peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Couldn't send request to peer 048cd781735346e9a1e53c48b25f0e38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d870ffae55284213b10a7256f864a501. This is attempt 1: this message will repeat every 5th retry.
I20250112 02:58:15.167383 24745 tablet_bootstrap.cc:654] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:15.167892 24751 consensus_queue.cc:260] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.179383 24175 heartbeater.cc:502] Master 127.20.217.189:40327 was elected leader, sending a full tablet report...
@ 0x7fa21a46b8d7 kudu::Thread::SuperviseThread() at ??:0
@ 0x468a00 __tsan_thread_start_func at ??:?
W20250112 02:58:15.184533 23848 catalog_manager.cc:5359] Ignoring report from unknown tablet 5b3da24feed4401aaa584995fb696586
W20250112 02:58:15.185076 23848 catalog_manager.cc:5359] Ignoring report from unknown tablet b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:15.188162 24751 ts_tablet_manager.cc:1428] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.040s user 0.007s sys 0.001s
I20250112 02:58:15.193672 23840 master_service.cc:946] ReplaceTablet: received request to replace tablet 112bccd320884506853e7b3870ca8440 from {username='slave'} at 127.0.0.1:58426
I20250112 02:58:15.200604 24751 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:15.204214 24745 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38: No bootstrap required, opened a new log
I20250112 02:58:15.204525 24746 ts_tablet_manager.cc:1918] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.204893 24746 log.cc:1198] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/5b3da24feed4401aaa584995fb696586
I20250112 02:58:15.205567 24746 ts_tablet_manager.cc:1939] T 5b3da24feed4401aaa584995fb696586 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:15.208034 24746 tablet_replica.cc:331] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:15.208712 24746 raft_consensus.cc:2238] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:58:15.209033 24747 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252: No bootstrap required, opened a new log
I20250112 02:58:15.209439 24747 ts_tablet_manager.cc:1397] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.000s
I20250112 02:58:15.209622 24746 raft_consensus.cc:2267] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.211979 24747 raft_consensus.cc:357] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.212734 24747 raft_consensus.cc:383] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.213007 24747 raft_consensus.cc:738] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Initialized, Role: FOLLOWER
I20250112 02:58:15.214982 24751 tablet_bootstrap.cc:654] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:15.215111 23834 catalog_manager.cc:4872] TS 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): tablet 5b3da24feed4401aaa584995fb696586 (table default.table_5bafe92abae84e2798e614a3069d4084 [id=cba9cafb990f42bfbdaa83dc8379d52a]) successfully deleted
I20250112 02:58:15.218777 24747 consensus_queue.cc:260] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.220579 23840 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 6ab49e495ba34dc29d914b277a872014: Sending DeleteTablet for 3 replicas of tablet 112bccd320884506853e7b3870ca8440
I20250112 02:58:15.221920 23840 catalog_manager.cc:6596] ReplaceTablet: tablet 112bccd320884506853e7b3870ca8440 deleted and replaced by tablet 5154cd0e702d416a859693e9db3cb25b
I20250112 02:58:15.224054 24393 tablet_service.cc:1514] Processing DeleteTablet for tablet 112bccd320884506853e7b3870ca8440 with delete_type TABLET_DATA_DELETED (replaced by tablet 5154cd0e702d416a859693e9db3cb25b) from {username='slave'} at 127.0.0.1:40634
I20250112 02:58:15.225422 24742 tablet_replica.cc:331] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: stopping tablet replica
I20250112 02:58:15.225854 24251 tablet_service.cc:1514] Processing DeleteTablet for tablet 112bccd320884506853e7b3870ca8440 with delete_type TABLET_DATA_DELETED (replaced by tablet 5154cd0e702d416a859693e9db3cb25b) from {username='slave'} at 127.0.0.1:47806
I20250112 02:58:15.226099 24742 raft_consensus.cc:2238] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:15.226538 24741 tablet_replica.cc:331] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: stopping tablet replica
I20250112 02:58:15.226735 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 112bccd320884506853e7b3870ca8440 with delete_type TABLET_DATA_DELETED (replaced by tablet 5154cd0e702d416a859693e9db3cb25b) from {username='slave'} at 127.0.0.1:53686
I20250112 02:58:15.227213 24742 raft_consensus.cc:2267] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.227236 24741 raft_consensus.cc:2238] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Raft consensus shutting down.
I20250112 02:58:15.227890 24741 pending_rounds.cc:62] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Trying to abort 1 pending ops.
I20250112 02:58:15.228173 24741 pending_rounds.cc:69] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7113321246936424448 op_type: NO_OP noop_request { }
I20250112 02:58:15.228569 24741 raft_consensus.cc:2883] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250112 02:58:15.228917 24741 raft_consensus.cc:2267] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.229367 24742 ts_tablet_manager.cc:1905] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.232002 24747 ts_tablet_manager.cc:1428] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250112 02:58:15.234764 24741 ts_tablet_manager.cc:1905] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7fa21baea6db start_thread at ??:0
I20250112 02:58:15.229368 24745 ts_tablet_manager.cc:1397] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.010s
W20250112 02:58:15.250397 23836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250112 02:58:15.251782 24745 raft_consensus.cc:357] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.252525 24745 raft_consensus.cc:383] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.252786 24745 raft_consensus.cc:738] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048cd781735346e9a1e53c48b25f0e38, State: Initialized, Role: FOLLOWER
@ 0x7fa216ecb71f clone at ??:0
I20250112 02:58:15.258416 24746 ts_tablet_manager.cc:1905] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.259564 24745 consensus_queue.cc:260] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.266474 24741 ts_tablet_manager.cc:1918] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.266846 24741 log.cc:1198] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-1/wal/wals/112bccd320884506853e7b3870ca8440
I20250112 02:58:15.267530 24741 ts_tablet_manager.cc:1939] T 112bccd320884506853e7b3870ca8440 P aff6bb94c7a14420a7540c9c198f0252: Deleting consensus metadata
I20250112 02:58:15.272778 24751 tablet_bootstrap.cc:492] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:15.273168 24751 ts_tablet_manager.cc:1397] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.007s
I20250112 02:58:15.275990 24742 ts_tablet_manager.cc:1918] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:15.276355 24742 log.cc:1198] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-2/wal/wals/112bccd320884506853e7b3870ca8440
I20250112 02:58:15.276948 24742 ts_tablet_manager.cc:1939] T 112bccd320884506853e7b3870ca8440 P 048cd781735346e9a1e53c48b25f0e38: Deleting consensus metadata
I20250112 02:58:15.279238 24745 ts_tablet_manager.cc:1428] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20250112 02:58:15.283823 24751 raft_consensus.cc:357] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:15.284495 24751 raft_consensus.cc:383] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.284749 24751 raft_consensus.cc:738] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:15.285346 24751 consensus_queue.cc:260] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } }
I20250112 02:58:15.287506 24751 ts_tablet_manager.cc:1428] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250112 02:58:15.288313 24751 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae: Bootstrap starting.
I20250112 02:58:15.295897 24746 ts_tablet_manager.cc:1918] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250112 02:58:15.296314 24746 log.cc:1198] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/d870ffae55284213b10a7256f864a501
I20250112 02:58:15.297017 24746 ts_tablet_manager.cc:1939] T d870ffae55284213b10a7256f864a501 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:15.298954 24751 tablet_bootstrap.cc:654] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae: Neither blocks nor log segments found. Creating new log.
I20250112 02:58:15.302040 24746 tablet_replica.cc:331] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: stopping tablet replica
I20250112 02:58:15.302893 24746 raft_consensus.cc:2238] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus shutting down.
I20250112 02:58:15.303426 24746 raft_consensus.cc:2267] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Raft consensus is shut down!
I20250112 02:58:15.306059 24746 ts_tablet_manager.cc:1905] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250112 02:58:15.311515 24751 tablet_bootstrap.cc:492] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae: No bootstrap required, opened a new log
I20250112 02:58:15.311964 24751 ts_tablet_manager.cc:1397] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.010s
I20250112 02:58:15.314837 24751 raft_consensus.cc:357] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.315663 24751 raft_consensus.cc:383] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 02:58:15.315984 24751 raft_consensus.cc:738] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Initialized, Role: FOLLOWER
I20250112 02:58:15.316656 24751 consensus_queue.cc:260] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.318986 24751 ts_tablet_manager.cc:1428] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250112 02:58:15.323933 24746 ts_tablet_manager.cc:1918] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250112 02:58:15.324357 24746 log.cc:1198] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting WAL directory at /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/ts-0/wal/wals/112bccd320884506853e7b3870ca8440
I20250112 02:58:15.325055 24746 ts_tablet_manager.cc:1939] T 112bccd320884506853e7b3870ca8440 P 4bea2270199d4f05ace161ff7f3fcfae: Deleting consensus metadata
I20250112 02:58:15.325513 24616 raft_consensus.cc:491] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:15.325938 24616 raft_consensus.cc:513] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.327818 24616 leader_election.cc:290] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:15.329270 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81458757d454d059ba307c6d228be9a" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae" is_pre_election: true
I20250112 02:58:15.329696 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81458757d454d059ba307c6d228be9a" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
I20250112 02:58:15.329854 24128 raft_consensus.cc:2463] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:15.330889 24206 leader_election.cc:304] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:15.330296 24414 raft_consensus.cc:2463] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 0.
I20250112 02:58:15.332427 24616 raft_consensus.cc:2798] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:15.332753 24616 raft_consensus.cc:491] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:15.333043 24616 raft_consensus.cc:3054] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.338129 24616 raft_consensus.cc:513] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.339581 24616 leader_election.cc:290] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Requested vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:15.340325 24128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81458757d454d059ba307c6d228be9a" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bea2270199d4f05ace161ff7f3fcfae"
I20250112 02:58:15.340826 24128 raft_consensus.cc:3054] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.341061 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e81458757d454d059ba307c6d228be9a" candidate_uuid: "aff6bb94c7a14420a7540c9c198f0252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:15.341574 24414 raft_consensus.cc:3054] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.347054 24128 raft_consensus.cc:2463] T e81458757d454d059ba307c6d228be9a P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:15.348021 24206 leader_election.cc:304] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:15.348873 24616 raft_consensus.cc:2798] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:15.349357 24616 raft_consensus.cc:695] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [term 1 LEADER]: Becoming Leader. State: Replica: aff6bb94c7a14420a7540c9c198f0252, State: Running, Role: LEADER
I20250112 02:58:15.349093 24414 raft_consensus.cc:2463] T e81458757d454d059ba307c6d228be9a P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff6bb94c7a14420a7540c9c198f0252 in term 1.
I20250112 02:58:15.351979 24616 consensus_queue.cc:237] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.466781 24761 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:15.467248 24761 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.469154 24761 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:15.471035 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:15.472060 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:15.472496 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:15.473343 24761 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:58:15.515138 24349 connection.cc:537] server connection from 127.0.0.1:40634 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.515923 24063 connection.cc:537] server connection from 127.0.0.1:53686 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.516196 24351 connection.cc:537] client connection to 127.20.217.189:40327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.516552 24474 connection.cc:537] client connection to 127.20.217.189:40327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.517824 24632 connection.cc:537] server connection from 127.0.0.1:43096 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.518179 24474 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.20.217.189:40327)
W20250112 02:58:15.518702 24208 connection.cc:537] server connection from 127.0.0.1:47806 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 02:58:15.519903 24754 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 02:58:15.517881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.217.189:40327 (local address 127.20.217.131:38047)
0112 02:58:15.519039 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0112 02:58:15.519223 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":314}
W20250112 02:58:15.520943 24063 connection.cc:537] client connection to 127.20.217.189:40327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:58:15.521878 24761 raft_consensus.cc:491] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:15.522337 24761 raft_consensus.cc:513] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
W20250112 02:58:15.524125 24474 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.20.217.189:40327)
I20250112 02:58:15.524117 24761 leader_election.cc:290] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:15.524786 24474 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.217.189:40327)
I20250112 02:58:15.527877 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc0b4256c94a9094c2b8f402056b2c" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38" is_pre_election: true
I20250112 02:58:15.528478 24414 raft_consensus.cc:2463] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 0.
W20250112 02:58:15.528854 24474 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.20.217.189:40327)
I20250112 02:58:15.529719 24062 leader_election.cc:304] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38, 4bea2270199d4f05ace161ff7f3fcfae; no voters:
I20250112 02:58:15.532048 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc0b4256c94a9094c2b8f402056b2c" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252" is_pre_election: true
I20250112 02:58:15.532665 24272 raft_consensus.cc:2463] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 0.
I20250112 02:58:15.534756 24761 raft_consensus.cc:2798] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 02:58:15.535038 24761 raft_consensus.cc:491] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 02:58:15.535305 24761 raft_consensus.cc:3054] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.541518 24761 raft_consensus.cc:513] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.544039 24272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc0b4256c94a9094c2b8f402056b2c" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff6bb94c7a14420a7540c9c198f0252"
I20250112 02:58:15.544571 24272 raft_consensus.cc:3054] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.551046 24272 raft_consensus.cc:2463] T 4fbc0b4256c94a9094c2b8f402056b2c P aff6bb94c7a14420a7540c9c198f0252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 1.
I20250112 02:58:15.552119 24060 leader_election.cc:304] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252; no voters:
I20250112 02:58:15.559410 24414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fbc0b4256c94a9094c2b8f402056b2c" candidate_uuid: "4bea2270199d4f05ace161ff7f3fcfae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048cd781735346e9a1e53c48b25f0e38"
I20250112 02:58:15.559974 24414 raft_consensus.cc:3054] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Advancing to term 1
I20250112 02:58:15.566179 24414 raft_consensus.cc:2463] T 4fbc0b4256c94a9094c2b8f402056b2c P 048cd781735346e9a1e53c48b25f0e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea2270199d4f05ace161ff7f3fcfae in term 1.
I20250112 02:58:15.567786 24761 leader_election.cc:290] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 election: Requested vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:15.568689 24761 raft_consensus.cc:2798] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 02:58:15.569682 24761 raft_consensus.cc:695] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [term 1 LEADER]: Becoming Leader. State: Replica: 4bea2270199d4f05ace161ff7f3fcfae, State: Running, Role: LEADER
I20250112 02:58:15.571054 24761 consensus_queue.cc:237] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:15.756481 24776 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6ab49e495ba34dc29d914b277a872014)
I20250112 02:58:15.756848 24776 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:15.758457 24776 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:15.767007 24633 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:15.769485 24633 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:15.769872 24633 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:15.770946 24776 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:16.167476 21350 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu with pid 24510
W20250112 02:58:16.199193 24473 connection.cc:537] client connection to 127.20.217.190:33757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 02:58:16.372838 24783 consensus_queue.cc:1035] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:16.389492 24788 consensus_queue.cc:1035] T e81458757d454d059ba307c6d228be9a P aff6bb94c7a14420a7540c9c198f0252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250112 02:58:16.570600 24784 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:16.570978 24784 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:16.572425 24784 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:16.574002 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:16.574443 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:16.574760 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:16.575310 24784 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:16.703346 24793 consensus_queue.cc:1035] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:58:16.717684 24800 consensus_queue.cc:1035] T 4fbc0b4256c94a9094c2b8f402056b2c P 4bea2270199d4f05ace161ff7f3fcfae [LEADER]: Connected to new peer: Peer: permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 02:58:17.097622 21350 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
/tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--webserver_interface=127.20.217.190
--webserver_port=46241
--builtin_ntp_servers=127.20.217.148:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--hive_metastore_uris=thrift://127.0.0.1:37187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250112 02:58:17.413749 24802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 02:58:17.414274 24802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 02:58:17.414724 24802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 02:58:17.445806 24802 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250112 02:58:17.446029 24802 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250112 02:58:17.446338 24802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 02:58:17.446610 24802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 02:58:17.446851 24802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 02:58:17.447073 24802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 02:58:17.482622 24802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.217.148:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:37187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.217.190:33757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.217.190
--webserver_port=46241
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 02:43:24 UTC on a633016ff5c9
build id 4136
TSAN enabled
I20250112 02:58:17.484014 24802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 02:58:17.485577 24802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 02:58:17.506389 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:17.509431 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 02:58:18.812028 24802 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 1.305s user 0.491s sys 0.810s
W20250112 02:58:18.812278 24802 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 1.305s user 0.491s sys 0.810s
I20250112 02:58:18.822049 24802 server_base.cc:1034] running on GCE node
W20250112 02:58:18.824257 24839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 02:58:18.825536 24802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 02:58:18.828071 24802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 02:58:18.829418 24802 hybrid_clock.cc:648] HybridClock initialized: now 1736650698829383 us; error 37 us; skew 500 ppm
I20250112 02:58:18.830179 24802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 02:58:18.842293 24802 webserver.cc:458] Webserver started at http://127.20.217.190:46241/ using document root <none> and password file <none>
I20250112 02:58:18.843241 24802 fs_manager.cc:362] Metadata directory not provided
I20250112 02:58:18.843451 24802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 02:58:18.852018 24802 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250112 02:58:18.858225 24844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 02:58:18.859432 24802 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250112 02:58:18.859732 24802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data,/tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
uuid: "98781ea331e8470ba27c6fc976a70be7"
format_stamp: "Formatted at 2025-01-12 02:57:58 on dist-test-slave-gppw"
I20250112 02:58:18.861950 24802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 02:58:18.935582 24802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 02:58:18.937014 24802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 02:58:18.937436 24802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 02:58:19.004344 24802 rpc_server.cc:307] RPC server started. Bound to: 127.20.217.190:33757
I20250112 02:58:19.004505 24895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.217.190:33757 every 8 connection(s)
I20250112 02:58:19.007099 24802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxVZiJz/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736650618888736-21350-0/minicluster-data/master-0/data/info.pb
I20250112 02:58:19.012212 21350 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskxVZiJz/build/tsan/bin/kudu as pid 24802
I20250112 02:58:19.019037 24896 sys_catalog.cc:263] Verifying existing consensus state
I20250112 02:58:19.026413 24896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap starting.
I20250112 02:58:19.050953 24896 log.cc:826] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Log is configured to *not* fsync() on all Append() calls
I20250112 02:58:19.071815 24176 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:19.132926 24896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=19 ignored=0} mutations{seen=34 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 02:58:19.133675 24896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: Bootstrap complete.
I20250112 02:58:19.150842 24896 raft_consensus.cc:357] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:19.152961 24896 raft_consensus.cc:738] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98781ea331e8470ba27c6fc976a70be7, State: Initialized, Role: FOLLOWER
I20250112 02:58:19.153710 24896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 3.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:19.155812 24902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } } }
I20250112 02:58:19.156435 24902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: This master's current role is: FOLLOWER
I20250112 02:58:19.156950 24896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 02:58:19.200336 24917 catalog_manager.cc:1260] Loaded cluster ID: 772ec94d997f43a8bcf9689121d3b747
I20250112 02:58:19.200639 24917 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: loading cluster ID for follower catalog manager: success
I20250112 02:58:19.210116 24917 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: acquiring CA information for follower catalog manager: success
I20250112 02:58:19.217195 24917 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250112 02:58:19.209188 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:19.238771 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:19.291965 24463 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:19.297788 24860 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" instance_seqno: 1736650689444137) as {username='slave'} at 127.20.217.131:52469; Asking this server to re-register.
I20250112 02:58:19.298976 24463 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:19.302152 24860 ts_manager.cc:194] Registered new tserver with Master: 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
I20250112 02:58:19.309571 24319 heartbeater.cc:346] Connected to a master server at 127.20.217.190:33757
I20250112 02:58:19.312562 24860 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" instance_seqno: 1736650687559117) as {username='slave'} at 127.20.217.130:39995; Asking this server to re-register.
I20250112 02:58:19.313505 24319 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:19.316298 24860 ts_manager.cc:194] Registered new tserver with Master: aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
I20250112 02:58:19.579468 24860 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" instance_seqno: 1736650685450797) as {username='slave'} at 127.20.217.129:41503; Asking this server to re-register.
I20250112 02:58:19.580626 24176 heartbeater.cc:463] Registering TS with master...
I20250112 02:58:19.583215 24860 ts_manager.cc:194] Registered new tserver with Master: 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241)
I20250112 02:58:19.704120 24923 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:19.704640 24923 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:19.706588 24923 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:19.708312 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:19.709110 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:19.709483 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:19.710060 24923 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:19.770696 24924 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:19.771147 24924 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:19.773397 24924 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:19.777472 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:19.781968 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:19.782421 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:19.782749 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:19.783345 24924 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:20.283826 24930 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:20.284384 24930 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:20.285931 24930 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:20.287475 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:20.287925 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:20.288316 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:20.289040 24930 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:20.497217 24934 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:20.497660 24934 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:20.499495 24934 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:20.519052 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:20.519712 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:20.520135 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:20.520826 24934 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:21.386423 21350 master-stress-test.cc:577] Killing test thread 1/10
I20250112 02:58:21.634728 24946 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:21.635186 24946 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:21.637033 24946 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:21.641765 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:21.642374 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:21.642668 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:21.643106 24946 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:23.116278 24966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:23.116641 24966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:23.117980 24966 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:23.122380 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:23.122956 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:23.123243 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:23.123731 24966 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:24.243597 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:24.243924 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:24.935928 24988 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:24.936308 24988 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:24.937798 24988 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:24.939410 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:24.939757 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:24.940032 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:24.940512 24988 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:25.628304 24998 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:25.628754 24998 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:25.630338 24998 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:25.632349 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250112 02:58:25.634578 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:25.635250 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:25.635517 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:25.636014 24998 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:28.563982 25032 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:28.564477 25032 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:28.566017 25032 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:28.571309 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:28.571805 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:28.572082 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:28.572651 25032 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:29.196594 25038 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:29.197117 25038 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:29.199298 25038 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:29.200875 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:29.201345 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:29.201617 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:29.202126 25038 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:29.248844 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:29.249222 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:32.080545 25085 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:32.080904 25085 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:32.082396 25085 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:32.084787 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 02:58:32.087316 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:32.087975 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:32.088400 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:32.089370 25085 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 02:58:32.326591 24892 debug-util.cc:398] Leaking SignalData structure 0x7b0800073060 after lost signal to thread 24811
W20250112 02:58:32.327157 24892 debug-util.cc:398] Leaking SignalData structure 0x7b080008ea00 after lost signal to thread 24895
I20250112 02:58:34.283591 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:34.283900 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:58:34.838788 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000df760 after lost signal to thread 24185
W20250112 02:58:34.839736 24313 debug-util.cc:398] Leaking SignalData structure 0x7b0800072120 after lost signal to thread 24316
I20250112 02:58:35.149868 25133 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:35.150373 25133 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:35.152725 25133 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:35.154449 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:35.155078 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:35.155375 24349 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:35.155910 25133 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:36.422354 25151 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:36.422741 25151 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:36.424188 25151 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:36.433676 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:36.435174 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:36.435647 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:36.436430 25151 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:36.780552 25148 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:36.781069 25148 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:36.782753 25148 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:36.784317 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:36.784770 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:36.785053 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:36.785583 25148 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:39.289696 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:39.290176 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:58:41.384616 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000d9b60 after lost signal to thread 24039
W20250112 02:58:41.385473 24169 debug-util.cc:398] Leaking SignalData structure 0x7b0800070e40 after lost signal to thread 24172
I20250112 02:58:44.294952 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:44.295266 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:48.553117 25333 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:48.553582 25333 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:58:48.555081 25333 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:58:48.557426 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 02:58:48.560115 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:58:48.560786 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:58:48.561139 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:58:48.561672 25333 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:58:49.301441 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:49.301887 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:49.835799 25355 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:49.836315 25355 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:49.838027 25355 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:49.839635 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:49.840092 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:49.840415 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:49.840898 25355 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:54.306563 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:54.306905 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:58:56.469092 25437 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:56.469669 25437 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:56.471784 25437 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:58:56.473788 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:58:56.474232 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:58:56.474560 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:56.475119 25437 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:57.868969 25470 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:58:57.869331 25470 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:58:57.870992 25470 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:58:57.872273 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:58:57.872731 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:58:57.873013 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:58:57.873521 25470 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:58:59.312465 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:58:59.312933 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:00.594491 25510 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:00.594974 25510 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:00.596537 25510 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:00.601094 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 02:59:00.603600 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:00.604136 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:00.604458 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:00.604990 25510 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:02.419039 25539 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:02.419384 25539 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:02.420711 25539 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:02.425408 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:02.425864 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:02.426136 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:02.426636 25539 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:04.317560 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:04.317862 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:06.224869 25600 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:06.225370 25600 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:06.227030 25600 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:59:06.229142 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:59:06.229854 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:59:06.230113 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:06.230608 25600 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:06.599316 25606 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:06.599670 25606 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:06.600982 25606 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:06.603228 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 02:59:06.605892 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:06.606747 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:06.607149 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:06.607846 25606 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:08.174525 25633 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:08.175276 25633 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:08.177062 25633 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:59:08.178740 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:59:08.179240 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:59:08.179572 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:08.180122 25633 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:09.323503 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:09.323925 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:59:09.940634 24175 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (112 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:10.118547 24317 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (128 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 02:59:10.162546 24461 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (126 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:12.912017 25702 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:12.912468 25702 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:12.913905 25702 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:12.916364 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 02:59:12.919292 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:12.919838 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:12.920099 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:12.920642 25702 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:14.328877 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:14.329217 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:15.365904 25742 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:15.366274 25742 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:15.367672 25742 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:15.372336 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:15.372802 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:15.373064 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:15.373565 25742 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:19.334757 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:19.335188 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:24.339941 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:24.340296 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:25.547526 25891 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:25.548027 25891 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:25.549875 25891 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:59:25.551687 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:59:25.552170 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:59:25.552481 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:25.553004 25891 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:25.869488 25899 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:25.869912 25899 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:25.871352 25899 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:25.873821 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 02:59:25.876865 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:25.877475 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:25.877758 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:25.878288 25899 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:26.220474 25894 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:26.220956 25894 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:26.223197 25894 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:59:26.224705 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:59:26.225154 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:59:26.225385 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:26.225828 25894 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:29.345556 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:29.345968 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:31.021216 25977 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:31.021642 25977 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:31.023138 25977 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:31.025545 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 02:59:31.028595 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:31.029078 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:31.029333 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:31.029853 25977 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:34.352770 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:34.353140 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:39.358726 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:39.359119 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:40.492323 26122 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:40.492897 26122 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:40.494711 26122 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:59:40.496567 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:59:40.497242 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:59:40.497813 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:40.498685 26122 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 02:59:40.984768 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000dede0 after lost signal to thread 24039
W20250112 02:59:40.985555 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4240 after lost signal to thread 24172
I20250112 02:59:41.985061 26143 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:41.985504 26143 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:41.987323 26143 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 02:59:41.988716 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 02:59:41.989105 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 02:59:41.989354 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:41.989836 26143 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:43.016794 26157 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:43.017247 26157 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 02:59:43.019034 26157 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 02:59:43.021829 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 02:59:43.024873 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 02:59:43.025434 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 02:59:43.025671 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 02:59:43.026130 26157 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 02:59:44.363425 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:44.363744 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:59:47.983606 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000e85e0 after lost signal to thread 24328
W20250112 02:59:47.984455 24457 debug-util.cc:398] Leaking SignalData structure 0x7b0800071320 after lost signal to thread 24460
I20250112 02:59:49.369772 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:49.370177 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 02:59:50.774448 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000db180 after lost signal to thread 24328
W20250112 02:59:50.775293 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000e88a0 after lost signal to thread 24460
I20250112 02:59:54.374581 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:54.374985 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 02:59:57.550077 26371 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 02:59:57.550593 26371 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 02:59:57.552866 26371 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 02:59:57.554677 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 02:59:57.555245 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 02:59:57.555522 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 02:59:57.556049 26371 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 02:59:59.380410 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 02:59:59.380864 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:00.759264 26420 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:00.759713 26420 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:00.761695 26420 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:00.763965 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:00:00.766688 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:00.767387 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:00.767659 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:00.768178 26420 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:00:00.900616 26426 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:00.901160 26426 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:00.903275 26426 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:00.904982 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:00.905442 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:00.905692 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:00.906239 26426 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 03:00:01.500238 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000dd280 after lost signal to thread 24185
W20250112 03:00:01.501101 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc640 after lost signal to thread 24316
I20250112 03:00:03.631191 26467 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:03.631639 26467 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:03.633473 26467 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:03.641614 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:03.642047 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:03.642290 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:03.642850 26467 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:00:04.385273 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:04.385609 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:04.467701 26473 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:04.468183 26473 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:04.469735 26473 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:04.471287 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:04.471724 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:04.471956 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:04.472430 26473 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:09.391463 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:09.391857 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:00:10.375561 24173 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (245 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:10.384034 26567 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:10.384482 26567 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:10.385973 26567 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:10.388108 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 03:00:10.390448 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:10.391155 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:10.391410 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:10.391881 26567 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 03:00:10.566519 24318 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (232 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:10.656643 24461 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (246 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:11.769539 26590 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:11.770084 26590 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:11.771849 26590 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:00:11.773311 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:00:11.773841 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:00:11.774142 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:11.774741 26590 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:14.396567 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:14.396919 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:14.582746 26631 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:14.583262 26631 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:14.584825 26631 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:14.586282 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:14.586786 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:14.587060 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:14.587782 26631 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:18.951133 26699 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:18.951586 26699 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:18.953394 26699 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:18.955858 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:00:18.958389 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:18.959002 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:18.959260 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:18.959734 26699 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:00:19.213752 24850 maintenance_manager.cc:418] P 98781ea331e8470ba27c6fc976a70be7: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033349
I20250112 03:00:19.290138 24849 maintenance_manager.cc:642] P 98781ea331e8470ba27c6fc976a70be7: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.040s sys 0.008s Metrics: {"bytes_written":11490,"cfile_init":1,"drs_written":1,"lbm_read_time_us":188,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2035,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":19,"thread_start_us":575,"threads_started":1,"wal-append.queue_time_us":867}
I20250112 03:00:19.292006 24850 maintenance_manager.cc:418] P 98781ea331e8470ba27c6fc976a70be7: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1178 bytes on disk
I20250112 03:00:19.293409 24849 maintenance_manager.cc:642] P 98781ea331e8470ba27c6fc976a70be7: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":210,"lbm_reads_lt_1ms":4}
I20250112 03:00:19.401229 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:19.401603 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:00:20.658643 24892 debug-util.cc:398] Leaking SignalData structure 0x7b08000808c0 after lost signal to thread 24811
W20250112 03:00:20.659202 24892 debug-util.cc:398] Leaking SignalData structure 0x7b080007ad00 after lost signal to thread 24895
I20250112 03:00:24.447193 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:24.447537 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:28.379433 26863 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:28.380092 26863 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:28.382392 26863 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:00:28.383949 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:00:28.384555 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:00:28.384827 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:28.385370 26863 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:29.452724 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:29.453109 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:30.440405 26891 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:30.440914 26891 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:30.443058 26891 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:30.444639 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:30.445079 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:30.445360 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:30.445869 26891 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:32.385206 26938 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:32.385689 26938 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:32.387367 26938 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:32.389382 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:00:32.391638 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:32.392303 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:32.392541 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:32.393016 26938 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:00:34.457727 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:34.458031 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:38.456696 27038 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:38.457217 27038 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:38.458941 27038 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:38.460479 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:38.460886 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:38.461165 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:38.461680 27038 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:39.463349 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:39.463786 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:44.468441 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:44.468739 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:00:46.296105 24892 debug-util.cc:398] Leaking SignalData structure 0x7b080009d240 after lost signal to thread 24811
W20250112 03:00:46.296629 24892 debug-util.cc:398] Leaking SignalData structure 0x7b08000b7680 after lost signal to thread 24895
I20250112 03:00:47.680792 27215 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:47.681355 27215 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:47.683308 27215 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:00:47.684944 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:00:47.685396 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:00:47.685688 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:47.686178 27215 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:49.495713 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:49.496133 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:50.755618 27270 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:50.756122 27270 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:50.757951 27270 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:50.760279 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:00:50.763321 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:50.763851 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:50.764173 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:50.764915 27270 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:00:51.332278 27279 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:51.332788 27279 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:00:51.334826 27279 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:00:51.336295 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:00:51.336792 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:00:51.337067 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:00:51.337592 27279 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:00:54.500643 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:54.500963 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:59.506378 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:00:59.506860 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:00:59.658401 27430 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:00:59.658937 27430 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:00:59.660586 27430 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:00:59.662580 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:00:59.666306 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:00:59.666756 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:00:59.667003 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:00:59.667538 27430 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 03:01:01.912405 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000dcd00 after lost signal to thread 24185
W20250112 03:01:01.913508 24313 debug-util.cc:398] Leaking SignalData structure 0x7b0800044520 after lost signal to thread 24316
I20250112 03:01:02.661208 27476 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:02.661751 27476 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:02.663518 27476 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:02.665062 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:02.665575 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:02.665848 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:02.666324 27476 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 03:01:03.909629 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000c53c0 after lost signal to thread 24328
W20250112 03:01:03.910542 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc020 after lost signal to thread 24460
I20250112 03:01:04.511404 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:04.511742 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:07.025584 27565 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:07.026103 27565 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:07.028348 27565 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:01:07.029820 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:01:07.030428 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:01:07.030755 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:07.031273 27565 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:09.517009 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:09.517416 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:01:10.793645 24175 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (352 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:10.982374 24317 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (368 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 03:01:11.032506 24461 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (366 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:11.055037 27638 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:11.055531 27638 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:11.057253 27638 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:01:11.058763 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:01:11.059320 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:01:11.059602 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:11.060137 27638 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:13.141034 27676 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:13.141461 27676 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:01:13.143208 27676 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:01:13.145431 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:01:13.147689 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:13.148379 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:13.148617 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:01:13.149075 27676 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:01:14.111649 27697 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:14.112006 27697 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:01:14.113530 27697 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:01:14.117864 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:14.118451 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:14.118785 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:01:14.119244 27697 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:01:14.521502 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:14.521847 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:19.527289 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:19.527705 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:21.681869 27836 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:21.682394 27836 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:21.684486 27836 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:21.686066 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:21.686583 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:21.686823 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:21.687315 27836 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:24.532265 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:24.532634 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:27.318957 27933 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:27.319502 27933 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:27.321540 27933 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:01:27.323309 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:01:27.323870 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:01:27.324151 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:27.324695 27933 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:28.584101 27956 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:28.584726 27956 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:28.586762 27956 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:28.588248 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:28.589192 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:28.589517 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:28.589989 27956 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:29.089669 27966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:29.090236 27966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:01:29.091899 27966 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:01:29.094671 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 03:01:29.097502 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:29.098032 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:29.098243 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:01:29.098765 27966 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:01:29.537061 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:29.537458 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:31.611197 28000 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:31.611580 28000 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:31.612931 28000 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:31.614156 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:31.614511 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:31.614784 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:31.615235 28000 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:34.542188 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:34.542527 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:36.513263 28088 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:36.513803 28088 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:36.515594 28088 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:36.517174 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:36.517827 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:36.518102 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:36.518626 28088 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:37.651024 28111 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:37.651569 28111 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:37.653301 28111 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:01:37.654841 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:01:37.655385 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:01:37.655689 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:37.656198 28111 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:39.547740 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:39.548141 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:01:39.789057 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6280 after lost signal to thread 24039
W20250112 03:01:39.789877 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000c26e0 after lost signal to thread 24172
I20250112 03:01:40.692579 28166 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:40.693066 28166 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:01:40.694634 28166 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:01:40.697069 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:01:40.699836 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:40.700539 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:40.700904 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:01:40.701508 28166 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:01:44.552691 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:44.553025 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:45.402740 28254 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:45.403240 28254 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:45.405289 28254 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:01:45.406854 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:01:45.407342 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:01:45.407636 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:45.408146 28254 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 03:01:47.111531 24313 debug-util.cc:398] Leaking SignalData structure 0x7b0800045b40 after lost signal to thread 24185
W20250112 03:01:47.113060 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000dcda0 after lost signal to thread 24316
I20250112 03:01:49.558382 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:49.558833 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:49.752216 28327 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:49.752763 28327 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:01:49.755306 28327 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:01:49.756754 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:01:49.757293 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:01:49.757555 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:01:49.758067 28327 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:01:54.563186 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:54.563519 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:01:56.516625 28446 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:01:56.517051 28446 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:01:56.518908 28446 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:01:56.521107 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:01:56.523790 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:01:56.524454 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:01:56.524689 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:01:56.525126 28446 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:01:59.569126 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:01:59.569543 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:00.254856 28516 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:00.255355 28516 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:00.256950 28516 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:00.258553 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:00.258967 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:00.259260 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:00.259760 28516 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:00.509400 28523 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:00.509963 28523 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:00.512363 28523 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:02:00.514042 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:02:00.514467 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:02:00.514766 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:00.515267 28523 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:04.480172 28590 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:04.480628 28590 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:04.482105 28590 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:04.484305 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:02:04.486788 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:04.487208 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:04.487437 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:04.487879 28590 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 03:02:04.504835 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000df5a0 after lost signal to thread 24039
W20250112 03:02:04.505631 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4e60 after lost signal to thread 24172
I20250112 03:02:04.573663 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:04.573997 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:02:08.671738 24892 debug-util.cc:398] Leaking SignalData structure 0x7b080008e060 after lost signal to thread 24811
W20250112 03:02:08.673048 24892 debug-util.cc:398] Leaking SignalData structure 0x7b0800052260 after lost signal to thread 24895
I20250112 03:02:09.602257 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:09.602659 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:02:11.219098 24175 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (472 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:11.386135 24318 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (472 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:11.395951 24462 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (472 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 03:02:14.607509 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:14.607982 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:14.673631 28772 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:14.674175 28772 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:14.675921 28772 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:02:14.677510 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:02:14.677953 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:02:14.678251 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:14.678826 28772 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:15.761663 28782 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:15.762181 28782 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:15.764300 28782 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:15.765725 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:15.766187 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:15.766417 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:15.766937 28782 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:19.613231 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:19.613600 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:02:21.727264 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4f20 after lost signal to thread 24039
W20250112 03:02:21.728081 24169 debug-util.cc:398] Leaking SignalData structure 0x7b08000e7d20 after lost signal to thread 24172
I20250112 03:02:23.756743 28929 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:23.757184 28929 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:23.759212 28929 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:23.761822 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:02:23.765174 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:23.765692 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:23.765995 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:23.766587 28929 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:02:24.617898 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:24.618213 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:29.623482 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:29.623873 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250112 03:02:30.529155 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000b1680 after lost signal to thread 24328
W20250112 03:02:30.529990 24457 debug-util.cc:398] Leaking SignalData structure 0x7b0800044a00 after lost signal to thread 24460
I20250112 03:02:30.784144 29051 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:30.784667 29051 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:30.786780 29051 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:02:30.788369 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:02:30.788843 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:02:30.789079 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:30.789525 29051 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:34.614012 29114 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:34.614557 29114 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:34.616675 29114 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:34.618136 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:34.618634 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:34.618882 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:34.619356 29114 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:34.628125 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:34.628415 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:39.164762 29200 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:39.165516 29200 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:39.167896 29200 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:02:39.169711 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:02:39.170266 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:02:39.170593 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:39.171097 29200 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:39.430059 29204 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:39.430543 29204 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:39.432618 29204 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:39.435154 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:02:39.437386 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:39.438047 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:39.438283 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:39.438778 29204 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:02:39.632323 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:39.632719 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:40.485769 29228 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:40.486119 29228 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:40.487705 29228 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:40.492026 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:40.492733 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:40.492976 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:40.493418 29228 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:02:44.636834 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:44.637135 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:45.022687 29304 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:45.023079 29304 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:45.024438 29304 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:45.027076 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 03:02:45.029626 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:45.030258 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:45.030537 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:45.030961 29304 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250112 03:02:45.271828 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000dcb40 after lost signal to thread 24185
W20250112 03:02:45.272697 24313 debug-util.cc:398] Leaking SignalData structure 0x7b08000d17a0 after lost signal to thread 24316
I20250112 03:02:45.688848 29312 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:45.689380 29312 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:45.691222 29312 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:45.692723 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:45.693248 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:45.693526 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:45.694036 29312 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:49.642665 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:49.643033 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:54.647604 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:54.647904 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:02:55.653285 29489 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:55.653757 29489 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:02:55.655400 29489 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:02:55.658120 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250112 03:02:55.660863 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:02:55.661300 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:02:55.661538 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:02:55.662015 29489 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:02:55.669023 29487 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:55.669548 29487 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:55.671780 29487 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:55.673976 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:55.674722 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:55.674970 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:55.675415 29487 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:55.695226 29494 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:55.695873 29494 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:55.698768 29494 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:02:55.699877 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:02:55.700439 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:02:55.700793 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:55.701406 29494 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:58.711583 29544 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:02:58.711966 29544 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:02:58.713551 29544 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:02:58.715350 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:02:58.716006 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:02:58.716284 24060 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:02:58.716727 29544 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:02:59.653192 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:02:59.653611 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:03:00.138267 29573 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:00.138782 29573 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:03:00.140275 29573 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:03:00.144974 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:03:00.145617 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:03:00.145987 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:03:00.146580 29573 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:03:04.658103 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:03:04.658411 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:03:05.344646 29665 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:05.345183 29665 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:03:05.346933 29665 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:03:05.348567 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:03:05.349022 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:03:05.349313 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:03:05.349839 29665 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 03:03:07.962603 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000df5e0 after lost signal to thread 24328
W20250112 03:03:07.963435 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000443c0 after lost signal to thread 24460
I20250112 03:03:09.465282 29725 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:09.465873 29725 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:03:09.467623 29725 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:03:09.469102 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:03:09.469586 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:03:09.469856 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:03:09.470355 29725 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:03:09.663640 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:03:09.664080 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:03:10.380518 29739 raft_consensus.cc:491] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:10.380909 29739 raft_consensus.cc:513] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:03:10.382375 29739 leader_election.cc:290] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691), 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845)
W20250112 03:03:10.383689 24060 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
W20250112 03:03:10.383977 24062 leader_election.cc:343] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 048cd781735346e9a1e53c48b25f0e38 (127.20.217.131:34845): Not found: Tablet not found: b6c6df7f45f04d46b788c13f1c3f2455
I20250112 03:03:10.384256 24062 leader_election.cc:304] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bea2270199d4f05ace161ff7f3fcfae; no voters: 048cd781735346e9a1e53c48b25f0e38, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:03:10.384713 29739 raft_consensus.cc:2743] T b6c6df7f45f04d46b788c13f1c3f2455 P 4bea2270199d4f05ace161ff7f3fcfae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250112 03:03:10.861054 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000da240 after lost signal to thread 24328
W20250112 03:03:10.861835 24457 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce120 after lost signal to thread 24460
W20250112 03:03:11.657117 24173 heartbeater.cc:643] Failed to heartbeat to 127.20.217.188:34169 (605 consecutive failures): Network error: Failed to ping master at 127.20.217.188:34169: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 03:03:11.773563 24318 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (592 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:03:11.812556 24462 heartbeater.cc:643] Failed to heartbeat to 127.20.217.189:40327 (592 consecutive failures): Network error: Failed to ping master at 127.20.217.189:40327: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
I20250112 03:03:13.689353 29781 raft_consensus.cc:491] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:13.689949 29781 raft_consensus.cc:513] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98781ea331e8470ba27c6fc976a70be7" member_type: VOTER last_known_addr { host: "127.20.217.190" port: 33757 } } peers { permanent_uuid: "6ab49e495ba34dc29d914b277a872014" member_type: VOTER last_known_addr { host: "127.20.217.189" port: 40327 } } peers { permanent_uuid: "2edd70f382cf4a44acb1db7b47eaf235" member_type: VOTER last_known_addr { host: "127.20.217.188" port: 34169 } }
I20250112 03:03:13.691525 29781 leader_election.cc:290] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327), 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169)
W20250112 03:03:13.694353 24848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20250112 03:03:13.697077 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6ab49e495ba34dc29d914b277a872014 (127.20.217.189:40327): Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111)
W20250112 03:03:13.697824 24848 leader_election.cc:336] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2edd70f382cf4a44acb1db7b47eaf235 (127.20.217.188:34169): Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
I20250112 03:03:13.698136 24848 leader_election.cc:304] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98781ea331e8470ba27c6fc976a70be7; no voters: 2edd70f382cf4a44acb1db7b47eaf235, 6ab49e495ba34dc29d914b277a872014
I20250112 03:03:13.698760 29781 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 98781ea331e8470ba27c6fc976a70be7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250112 03:03:14.184964 29792 raft_consensus.cc:491] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 03:03:14.185467 29792 raft_consensus.cc:513] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea2270199d4f05ace161ff7f3fcfae" member_type: VOTER last_known_addr { host: "127.20.217.129" port: 35241 } } peers { permanent_uuid: "aff6bb94c7a14420a7540c9c198f0252" member_type: VOTER last_known_addr { host: "127.20.217.130" port: 38691 } } peers { permanent_uuid: "048cd781735346e9a1e53c48b25f0e38" member_type: VOTER last_known_addr { host: "127.20.217.131" port: 34845 } }
I20250112 03:03:14.187187 29792 leader_election.cc:290] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241), aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691)
W20250112 03:03:14.188997 24349 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bea2270199d4f05ace161ff7f3fcfae (127.20.217.129:35241): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
W20250112 03:03:14.189416 24348 leader_election.cc:343] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer aff6bb94c7a14420a7540c9c198f0252 (127.20.217.130:38691): Not found: Tablet not found: fba5ffea5d5f4bf1a734eccee2cfc3c8
I20250112 03:03:14.189774 24348 leader_election.cc:304] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048cd781735346e9a1e53c48b25f0e38; no voters: 4bea2270199d4f05ace161ff7f3fcfae, aff6bb94c7a14420a7540c9c198f0252
I20250112 03:03:14.190428 29792 raft_consensus.cc:2743] T fba5ffea5d5f4bf1a734eccee2cfc3c8 P 048cd781735346e9a1e53c48b25f0e38 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 03:03:14.668918 24802 master_runner.cc:186] Error getting master registration for 127.20.217.189:40327: Network error: Client connection negotiation failed: client connection to 127.20.217.189:40327: connect: Connection refused (error 111),
I20250112 03:03:14.669312 24802 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250112 03:03:15.429602 24850 maintenance_manager.cc:418] P 98781ea331e8470ba27c6fc976a70be7: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1178 bytes on disk
I20250112 03:03:15.440213 24849 maintenance_manager.cc:642] P 98781ea331e8470ba27c6fc976a70be7: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.006s sys 0.004s Metrics: {"dirs.queue_time_us":593,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":216,"thread_start_us":744,"threads_started":2,"wal-append.queue_time_us":672}
W20250112 03:03:15.640461 24474 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 03:03:15.640877 24474 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 03:03:15.641093 24474 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
W20250112 03:03:15.641402 24474 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
F20250112 03:03:15.652489 24483 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20250112 03:03:15.652694 24485 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
*** Aborted at 1736650995 (unix time) try "date -d @1736650995" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005366) received by PID 21350 (TID 0x7f29acbfa700) from PID 21350; stack trace: ***
@ 0x2dac70 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2dd153 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f29c0456980 (unknown) at ??:0
@ 0x7f29bb66ffb7 gsignal at ??:0
*** Check failure stack trace: ***
@ 0x7f29bb671921 abort at ??:0
@ 0x2e0c66 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f29be72ada4 google::LogMessage::SendToLog() at ??:0
@ 0x7f29be72b950 google::LogMessage::Flush() at ??:0
@ 0x7f29be730a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35d396 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38abbe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x38aa71 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
F20250112 03:03:15.652694 24485 master-stress-test.cc:351] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)F20250112 03:03:15.663323 24480 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_ebcfbc2fbb6442918a4444cc0bcbab4a on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.20.217.190:33757,127.20.217.189:40327,127.20.217.188:34169, num_attempts: 769) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.20.217.188:34169: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x38a1bb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dab30 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f29c044b6db start_thread at ??:0
==================
WARNING: ThreadSanitizer: data race (pid=21350)
Atomic write of size 8 at 0x7f29be9568b8 by thread T28:
#0 __tsan_atomic64_compare_exchange_val /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cpp:850 (master-stress-test+0x334c58)
#1 unsigned long* google::glog_internal_namespace_::sync_val_compare_and_swap<unsigned long*>(unsigned long**, unsigned long*, unsigned long*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/utilities.h:172:10 (libglog.so.1+0x2b285)
#2 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/signalhandler.cc:296:7 (libglog.so.1+0x2b285)
#3 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1911 (master-stress-test+0x2dac6f)
#4 google::LogMessage::Fail() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1946:3 (libglog.so.1+0x18da3)
#5 google::LogMessage::SendToLog() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1920:5 (libglog.so.1+0x18da3)
#6 google::LogMessage::Flush() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1777:5 (libglog.so.1+0x1994f)
#7 google::LogMessageFatal::~LogMessageFatal() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:2557:5 (libglog.so.1+0x1ea8a)
#8 kudu::MasterStressTest::CreateTableThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc (master-stress-test+0x35cda3)
#9 decltype(*(std::__1::forward<kudu::MasterStressTest_Test_Test*>(fp0)).*fp()) std::__1::__invoke<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840:1 (master-stress-test+0x38abbd)
#10 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>&, std::__1::__tuple_indices<2ul>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:280:5 (master-stress-test+0x38aa70)
#11 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*> >(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:291:5 (master-stress-test+0x38a1ba)
Previous read of size 8 at 0x7f29be9568b8 by thread T33:
#0 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/signalhandler.cc:302:38 (libglog.so.1+0x2ca90)
#1 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1911 (master-stress-test+0x2dac6f)
#2 google::LogMessage::Fail() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1946:3 (libglog.so.1+0x18da3)
#3 google::LogMessage::SendToLog() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1920:5 (libglog.so.1+0x18da3)
#4 google::LogMessage::Flush() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:1777:5 (libglog.so.1+0x1994f)
#5 google::LogMessageFatal::~LogMessageFatal() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/logging.cc:2557:5 (libglog.so.1+0x1ea8a)
#6 kudu::MasterStressTest::DeleteTableThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:351:7 (master-stress-test+0x35d5bd)
#7 decltype(*(std::__1::forward<kudu::MasterStressTest_Test_Test*>(fp0)).*fp()) std::__1::__invoke<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840:1 (master-stress-test+0x38abbd)
#8 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>&, std::__1::__tuple_indices<2ul>) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:280:5 (master-stress-test+0x38aa70)
#9 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*> >(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:291:5 (master-stress-test+0x38a1ba)
Location is global 'google::(anonymous namespace)::g_entered_thread_id_pointer' of size 8 at 0x7f29be9568b8 (libglog.so.1+0x0000002448b8)
Thread T28 (tid=24480, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (master-stress-test+0x2de5e5)
#1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__threading_support:504:10 (master-stress-test+0x38a14b)
#2 std::__1::thread::thread<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:307:16 (master-stress-test+0x389ea7)
#3 void std::__1::allocator<std::__1::thread>::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1809:31 (master-stress-test+0x389dac)
#4 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::__construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (master-stress-test+0x389d43)
#5 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (master-stress-test+0x389c83)
#6 void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__emplace_back_slow_path<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1670:5 (master-stress-test+0x389abe)
#7 std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::emplace_back<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1690:9 (master-stress-test+0x35c80a)
#8 kudu::MasterStressTest_Test_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:557:13 (master-stress-test+0x35b676)
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#11 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#12 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#13 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#17 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#18 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (master-stress-test+0x38cf8b)
#19 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (master-stress-test+0x38be8c)
Thread T33 (tid=24485, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (master-stress-test+0x2de5e5)
#1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__threading_support:504:10 (master-stress-test+0x38a14b)
#2 std::__1::thread::thread<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*, void>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:307:16 (master-stress-test+0x389ea7)
#3 void std::__1::allocator<std::__1::thread>::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1809:31 (master-stress-test+0x389dac)
#4 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::__construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::integral_constant<bool, true>, std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1687:21 (master-stress-test+0x389d43)
#5 void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::construct<std::__1::thread, void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1538:14 (master-stress-test+0x389c83)
#6 void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__construct_one_at_end<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:926:5 (master-stress-test+0x38998e)
#7 std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::emplace_back<void (kudu::MasterStressTest::*)(), kudu::MasterStressTest_Test_Test*>(void (kudu::MasterStressTest::*&&)(), kudu::MasterStressTest_Test_Test*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/vector:1687:9 (master-stress-test+0x35c7fa)
#8 kudu::MasterStressTest_Test_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:563:13 (master-stress-test+0x35b756)
#9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#11 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#12 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#13 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#14 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#17 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#18 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (master-stress-test+0x38cf8b)
#19 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:109:10 (master-stress-test+0x38be8c)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/glog-0.6.0/src/utilities.h:172:10 in unsigned long* google::glog_internal_namespace_::sync_val_compare_and_swap<unsigned long*>(unsigned long**, unsigned long*, unsigned long*)
==================
@ 0x7f29bb75271f clone at ??:0