Note: This is test shard 2 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/1
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250115 20:50:23.044302 29263 test_util.cc:274] Using random seed: -745771356
2025-01-15T20:50:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-15T20:50:23Z Disabled control of system clock
I20250115 20:50:23.076325 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.254:45409
--webserver_interface=127.28.147.254
--webserver_port=0
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250115 20:50:23.382736 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:23.383033 29271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:23.383119 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:23.390671 29271 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250115 20:50:23.390767 29271 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250115 20:50:23.390810 29271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 20:50:23.390857 29271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:23.390901 29271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 20:50:23.390959 29271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 20:50:23.399785 29271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.254:45409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.147.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:23.401283 29271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:23.402894 29271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:23.413249 29276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:23.413897 29279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:23.414534 29277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:23.415223 29271 server_base.cc:1034] running on GCE node
I20250115 20:50:23.415993 29271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:23.417404 29271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:23.418607 29271 hybrid_clock.cc:648] HybridClock initialized: now 1736974223418547 us; error 34 us; skew 500 ppm
I20250115 20:50:23.418952 29271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:23.421284 29271 webserver.cc:458] Webserver started at http://127.28.147.254:40431/ using document root <none> and password file <none>
I20250115 20:50:23.422011 29271 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:23.422127 29271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:23.422418 29271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:23.424971 29271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data/instance:
uuid: "43a8f945f14d4739b9fb165b5a945ab7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.425627 29271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal/instance:
uuid: "43a8f945f14d4739b9fb165b5a945ab7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.430588 29271 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250115 20:50:23.433616 29285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:23.434859 29271 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250115 20:50:23.435009 29271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal
uuid: "43a8f945f14d4739b9fb165b5a945ab7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.435186 29271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:23.468858 29271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:23.469702 29271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:23.469992 29271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:23.488294 29271 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.254:45409
I20250115 20:50:23.488320 29336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.254:45409 every 8 connection(s)
I20250115 20:50:23.489743 29271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/data/info.pb
I20250115 20:50:23.491793 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29271
I20250115 20:50:23.492120 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-0/wal/instance
I20250115 20:50:23.494587 29337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:23.495487 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.253:46053
--webserver_interface=127.28.147.253
--webserver_port=0
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 20:50:23.502362 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:23.518039 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
W20250115 20:50:23.518925 29288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
W20250115 20:50:23.520188 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.253:46053: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
I20250115 20:50:23.569628 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } attempt: 1
W20250115 20:50:23.571491 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.253:46053: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
I20250115 20:50:23.639745 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } attempt: 2
W20250115 20:50:23.641482 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.253:46053: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
I20250115 20:50:23.732729 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } attempt: 3
W20250115 20:50:23.734640 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.253:46053: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
W20250115 20:50:23.814615 29339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:23.814932 29339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:23.815078 29339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:23.822799 29339 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250115 20:50:23.822928 29339 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250115 20:50:23.823005 29339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 20:50:23.823091 29339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:23.823166 29339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 20:50:23.823258 29339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 20:50:23.833160 29339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.253:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.147.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:23.834848 29339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:23.836632 29339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:23.847241 29351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:23.847920 29339 server_base.cc:1034] running on GCE node
W20250115 20:50:23.848455 29349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:23.848889 29347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:23.849714 29339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:23.851111 29339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:23.852326 29339 hybrid_clock.cc:648] HybridClock initialized: now 1736974223852265 us; error 38 us; skew 500 ppm
I20250115 20:50:23.852686 29339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:23.854799 29339 webserver.cc:458] Webserver started at http://127.28.147.253:45853/ using document root <none> and password file <none>
I20250115 20:50:23.855509 29339 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:23.855652 29339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:23.855942 29339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:23.858536 29339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data/instance:
uuid: "af9e84ab837a4c4face479825e0bded7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.859218 29339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal/instance:
uuid: "af9e84ab837a4c4face479825e0bded7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.863999 29339 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250115 20:50:23.867098 29357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:23.868232 29339 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250115 20:50:23.868384 29339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal
uuid: "af9e84ab837a4c4face479825e0bded7"
format_stamp: "Formatted at 2025-01-15 20:50:23 on dist-test-slave-2pn0"
I20250115 20:50:23.868613 29339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:23.877861 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } attempt: 4
W20250115 20:50:23.879482 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.253:46053: Network error: Client connection negotiation failed: client connection to 127.28.147.253:46053: connect: Connection refused (error 111)
I20250115 20:50:23.894619 29339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:23.895452 29339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:23.895740 29339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:23.912099 29339 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.253:46053
I20250115 20:50:23.912122 29409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.253:46053 every 8 connection(s)
I20250115 20:50:23.913445 29339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/data/info.pb
I20250115 20:50:23.917061 29410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:23.922398 29410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:23.923417 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29339
I20250115 20:50:23.923576 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-1/wal/instance
I20250115 20:50:23.924793 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.252:38139
--webserver_interface=127.28.147.252
--webserver_port=0
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 20:50:23.935846 29410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:23.945549 29410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } has no permanent_uuid. Determining permanent_uuid...
W20250115 20:50:23.946658 29359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
W20250115 20:50:23.948081 29410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:23.997445 29410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 1
W20250115 20:50:23.999358 29410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.067621 29410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 2
W20250115 20:50:24.069470 29410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.160739 29410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 3
W20250115 20:50:24.162617 29410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.178718 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } attempt: 5
I20250115 20:50:24.184877 29337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } has no permanent_uuid. Determining permanent_uuid...
W20250115 20:50:24.186704 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.237895 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 1
W20250115 20:50:24.239722 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
W20250115 20:50:24.244689 29413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:24.244969 29413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:24.245059 29413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:24.252255 29413 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250115 20:50:24.252333 29413 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250115 20:50:24.252370 29413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 20:50:24.252413 29413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:24.252447 29413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 20:50:24.252542 29413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 20:50:24.261515 29413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.147.252:38139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.147.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:24.263005 29413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:24.264526 29413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:24.274571 29421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:24.275326 29420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:24.275928 29423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:24.276739 29413 server_base.cc:1034] running on GCE node
I20250115 20:50:24.277474 29413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:24.278739 29413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:24.279958 29413 hybrid_clock.cc:648] HybridClock initialized: now 1736974224279903 us; error 32 us; skew 500 ppm
I20250115 20:50:24.280247 29413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:24.282358 29413 webserver.cc:458] Webserver started at http://127.28.147.252:36803/ using document root <none> and password file <none>
I20250115 20:50:24.283066 29413 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:24.283162 29413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:24.283407 29413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:24.286145 29413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data/instance:
uuid: "1195eca7127442e7a2c5a9e4df7b679a"
format_stamp: "Formatted at 2025-01-15 20:50:24 on dist-test-slave-2pn0"
I20250115 20:50:24.286803 29413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal/instance:
uuid: "1195eca7127442e7a2c5a9e4df7b679a"
format_stamp: "Formatted at 2025-01-15 20:50:24 on dist-test-slave-2pn0"
I20250115 20:50:24.291396 29413 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20250115 20:50:24.294191 29429 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:24.295300 29413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250115 20:50:24.295418 29413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal
uuid: "1195eca7127442e7a2c5a9e4df7b679a"
format_stamp: "Formatted at 2025-01-15 20:50:24 on dist-test-slave-2pn0"
I20250115 20:50:24.295559 29413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:24.305850 29410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 4
W20250115 20:50:24.307523 29410 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.307942 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 2
W20250115 20:50:24.309470 29337 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.147.252:38139: Network error: Client connection negotiation failed: client connection to 127.28.147.252:38139: connect: Connection refused (error 111)
I20250115 20:50:24.320569 29413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:24.321455 29413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:24.321725 29413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:24.338181 29413 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.252:38139
I20250115 20:50:24.338208 29481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.252:38139 every 8 connection(s)
I20250115 20:50:24.339428 29413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/data/info.pb
I20250115 20:50:24.342552 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29413
I20250115 20:50:24.342777 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/master-2/wal/instance
I20250115 20:50:24.343184 29482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:24.349974 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:24.362113 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:24.368904 29482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } has no permanent_uuid. Determining permanent_uuid...
I20250115 20:50:24.382015 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Bootstrap starting.
I20250115 20:50:24.385305 29482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:24.386364 29482 log.cc:826] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Log is configured to *not* fsync() on all Append() calls
I20250115 20:50:24.388907 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: No bootstrap required, opened a new log
I20250115 20:50:24.395128 29482 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.395406 29482 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:24.395524 29482 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1195eca7127442e7a2c5a9e4df7b679a, State: Initialized, Role: FOLLOWER
I20250115 20:50:24.396126 29482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.396924 29491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.397176 29491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:24.397912 29482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: configured and running, proceeding with master startup.
W20250115 20:50:24.407470 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.407596 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.404891 29413 master_runner.cc:186] Error getting master registration for 127.28.147.254:45409: OK, instance_id { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" instance_seqno: 1736974223483144 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250115 20:50:24.411514 29413 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250115 20:50:24.415771 29337 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 3
W20250115 20:50:24.418941 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.419070 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.425853 29337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Bootstrap starting.
I20250115 20:50:24.429754 29337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Neither blocks nor log segments found. Creating new log.
W20250115 20:50:24.430449 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.430553 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.431026 29337 log.cc:826] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Log is configured to *not* fsync() on all Append() calls
I20250115 20:50:24.433315 29337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: No bootstrap required, opened a new log
I20250115 20:50:24.439200 29337 raft_consensus.cc:357] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.439538 29337 raft_consensus.cc:383] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:24.439677 29337 raft_consensus.cc:738] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a8f945f14d4739b9fb165b5a945ab7, State: Initialized, Role: FOLLOWER
I20250115 20:50:24.440289 29337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.441242 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.441576 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: FOLLOWER
W20250115 20:50:24.441865 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.441990 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.442461 29337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: configured and running, proceeding with master startup.
W20250115 20:50:24.453243 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.453372 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.453435 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.453531 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.453950 29271 master_runner.cc:186] Error getting master registration for 127.28.147.253:46053: OK, instance_id { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" instance_seqno: 1736974223907807 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250115 20:50:24.462215 29271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250115 20:50:24.464816 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.464869 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.464942 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.465005 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.472671 29291 tablet.cc:2367] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250115 20:50:24.476279 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.476284 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.476394 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.476563 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.488070 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.488315 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.488605 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.488709 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.499692 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.499830 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.500056 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.500212 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.511096 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.511323 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.511488 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.511590 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.522795 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.522801 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.522943 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.523059 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.534494 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.534494 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.534655 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.534754 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.546289 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.546332 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.546448 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.546461 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.557947 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.557955 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.558099 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.558166 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.569612 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.569624 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.569761 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.569882 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.573694 29435 tablet.cc:2367] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 20:50:24.576643 29491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 20:50:24.576917 29491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.578670 29491 leader_election.cc:290] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053)
I20250115 20:50:24.579684 29312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7" is_pre_election: true
I20250115 20:50:24.580338 29312 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1195eca7127442e7a2c5a9e4df7b679a in term 0.
I20250115 20:50:24.581125 29432 leader_election.cc:304] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1195eca7127442e7a2c5a9e4df7b679a, 43a8f945f14d4739b9fb165b5a945ab7; no voters:
I20250115 20:50:24.581689 29491 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 20:50:24.581832 29491 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 20:50:24.581940 29491 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 0 FOLLOWER]: Advancing to term 1
W20250115 20:50:24.582170 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.582274 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.585616 29491 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.586813 29312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7"
I20250115 20:50:24.587044 29312 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 0 FOLLOWER]: Advancing to term 1
W20250115 20:50:24.589885 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.590044 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.590380 29312 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1195eca7127442e7a2c5a9e4df7b679a in term 1.
I20250115 20:50:24.590984 29432 leader_election.cc:304] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1195eca7127442e7a2c5a9e4df7b679a, 43a8f945f14d4739b9fb165b5a945ab7; no voters:
I20250115 20:50:24.591939 29518 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 FOLLOWER]: Leader election won for term 1
W20250115 20:50:24.593760 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.593869 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.594040 29491 leader_election.cc:290] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 1 election: Requested vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053)
I20250115 20:50:24.594585 29518 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 LEADER]: Becoming Leader. State: Replica: 1195eca7127442e7a2c5a9e4df7b679a, State: Running, Role: LEADER
I20250115 20:50:24.595749 29518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
W20250115 20:50:24.603202 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.603331 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.603471 29491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1195eca7127442e7a2c5a9e4df7b679a. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.603771 29491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: LEADER
W20250115 20:50:24.605286 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.605401 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.606720 29410 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } attempt: 5
W20250115 20:50:24.614408 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.614507 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.616851 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.616961 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.621760 29410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: Bootstrap starting.
W20250115 20:50:24.625550 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.625705 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.629135 29410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:24.635128 29410 log.cc:826] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: Log is configured to *not* fsync() on all Append() calls
W20250115 20:50:24.635553 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.635672 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.636885 29502 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.636986 29502 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.646988 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.647192 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.647193 29521 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 20:50:24.650110 29410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: No bootstrap required, opened a new log
I20250115 20:50:24.653316 29521 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250115 20:50:24.658273 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.658434 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.667781 29410 raft_consensus.cc:357] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:24.668160 29410 raft_consensus.cc:383] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:24.668326 29410 raft_consensus.cc:738] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9e84ab837a4c4face479825e0bded7, State: Initialized, Role: FOLLOWER
I20250115 20:50:24.669118 29410 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
W20250115 20:50:24.669708 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.669806 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.678119 29312 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Refusing update from remote peer 1195eca7127442e7a2c5a9e4df7b679a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 20:50:24.679210 29518 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 20:50:24.681217 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.681322 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.686502 29385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7" is_pre_election: true
I20250115 20:50:24.686514 29384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7"
I20250115 20:50:24.687240 29385 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1195eca7127442e7a2c5a9e4df7b679a in term 0.
I20250115 20:50:24.686527 29410 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 20:50:24.687810 29383 raft_consensus.cc:3054] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 0 FOLLOWER]: Advancing to term 1
I20250115 20:50:24.691731 29383 raft_consensus.cc:1270] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 1 FOLLOWER]: Refusing update from remote peer 1195eca7127442e7a2c5a9e4df7b679a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 20:50:24.692736 29518 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 20:50:24.693147 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.693562 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
W20250115 20:50:24.718377 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.718556 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.721074 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1195eca7127442e7a2c5a9e4df7b679a. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.721417 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:24.726435 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1195eca7127442e7a2c5a9e4df7b679a. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.726732 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:24.727838 29505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.728058 29505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:24.728401 29523 mvcc.cc:204] Tried to move back new op lower bound from 7114646424265236480 to 7114646423955820544. Current Snapshot: MvccSnapshot[applied={T|T < 7114646424265236480}]
W20250115 20:50:24.730377 29516 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 20:50:24.730504 29516 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.742342 29516 catalog_manager.cc:1260] Loaded cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:24.742475 29516 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: loading cluster ID for follower catalog manager: success
W20250115 20:50:24.743572 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.743698 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.755194 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.755321 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.755841 29524 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.756192 29524 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:24.762326 29491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.762696 29491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: LEADER
I20250115 20:50:24.763128 29491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:24.763388 29491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: LEADER
W20250115 20:50:24.766741 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.766868 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.767141 29521 catalog_manager.cc:1348] Generated new cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:24.767242 29521 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
W20250115 20:50:24.781145 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.781303 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.792959 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.793128 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.802579 29543 catalog_manager.cc:1260] Loaded cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:24.802776 29543 catalog_manager.cc:1553] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: loading cluster ID for follower catalog manager: success
W20250115 20:50:24.804327 29543 catalog_manager.cc:1577] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.804600 29543 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.804764 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.804891 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.816223 29543 catalog_manager.cc:1577] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.816445 29543 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.817266 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.817386 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.828199 29543 catalog_manager.cc:1577] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.828344 29543 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.828998 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.829128 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.839830 29543 catalog_manager.cc:1577] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.839972 29543 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.843801 29516 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250115 20:50:24.843924 29516 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.845953 29521 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 20:50:24.847774 29521 catalog_manager.cc:1505] Loading token signing keys...
I20250115 20:50:24.853964 29543 catalog_manager.cc:1575] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: acquiring CA information for follower catalog manager: success
W20250115 20:50:24.855325 29543 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.858141 29516 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: acquiring CA information for follower catalog manager: success
W20250115 20:50:24.859534 29516 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.867002 29543 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.872548 29516 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20250115 20:50:24.878518 29543 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250115 20:50:24.886300 29516 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 20:50:24.886688 29521 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Generated new TSK 0
I20250115 20:50:24.887575 29521 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 20:50:24.890774 29543 catalog_manager.cc:1603] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 20:50:24.919835 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.147.193:0
--local_ip_for_outbound_sockets=127.28.147.193
--webserver_interface=127.28.147.193
--webserver_port=0
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250115 20:50:25.308471 29545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:25.308943 29545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:25.309127 29545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:25.317649 29545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:25.317929 29545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.147.193
I20250115 20:50:25.328210 29545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.147.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.147.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.147.193
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:25.330057 29545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:25.332149 29545 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:25.346230 29553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:25.346761 29550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:25.347430 29551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:25.348173 29545 server_base.cc:1034] running on GCE node
I20250115 20:50:25.348868 29545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:25.349907 29545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:25.351147 29545 hybrid_clock.cc:648] HybridClock initialized: now 1736974225351075 us; error 46 us; skew 500 ppm
I20250115 20:50:25.351456 29545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:25.355502 29545 webserver.cc:458] Webserver started at http://127.28.147.193:36733/ using document root <none> and password file <none>
I20250115 20:50:25.356653 29545 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:25.356817 29545 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:25.357214 29545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:25.361243 29545 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data/instance:
uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.362303 29545 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/instance:
uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.368304 29545 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 20:50:25.371913 29559 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:25.373428 29545 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 20:50:25.373602 29545 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal
uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.373844 29545 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:25.382265 29564 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1195eca7127442e7a2c5a9e4df7b679a)
I20250115 20:50:25.382459 29564 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.384088 29565 raft_consensus.cc:491] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1195eca7127442e7a2c5a9e4df7b679a)
I20250115 20:50:25.384354 29565 raft_consensus.cc:513] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.384533 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7" is_pre_election: true
I20250115 20:50:25.384965 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a8f945f14d4739b9fb165b5a945ab7 in term 1.
I20250115 20:50:25.384991 29564 leader_election.cc:290] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053), 1195eca7127442e7a2c5a9e4df7b679a (127.28.147.252:38139)
I20250115 20:50:25.384914 29457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1195eca7127442e7a2c5a9e4df7b679a" is_pre_election: true
I20250115 20:50:25.385594 29288 leader_election.cc:304] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a8f945f14d4739b9fb165b5a945ab7, af9e84ab837a4c4face479825e0bded7; no voters:
I20250115 20:50:25.386263 29564 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250115 20:50:25.386386 29564 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1195eca7127442e7a2c5a9e4df7b679a)
I20250115 20:50:25.386497 29564 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 1 FOLLOWER]: Advancing to term 2
I20250115 20:50:25.386557 29311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af9e84ab837a4c4face479825e0bded7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7" is_pre_election: true
I20250115 20:50:25.386935 29565 leader_election.cc:290] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), 1195eca7127442e7a2c5a9e4df7b679a (127.28.147.252:38139)
I20250115 20:50:25.387176 29457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "af9e84ab837a4c4face479825e0bded7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1195eca7127442e7a2c5a9e4df7b679a" is_pre_election: true
I20250115 20:50:25.390162 29564 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.390666 29311 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af9e84ab837a4c4face479825e0bded7 in current term 2: Already voted for candidate 43a8f945f14d4739b9fb165b5a945ab7 in this term.
I20250115 20:50:25.390971 29564 leader_election.cc:290] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 2 election: Requested vote from peers af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053), 1195eca7127442e7a2c5a9e4df7b679a (127.28.147.252:38139)
I20250115 20:50:25.391567 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7"
I20250115 20:50:25.391772 29383 raft_consensus.cc:3054] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 1 FOLLOWER]: Advancing to term 2
I20250115 20:50:25.391842 29360 leader_election.cc:304] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af9e84ab837a4c4face479825e0bded7; no voters: 1195eca7127442e7a2c5a9e4df7b679a, 43a8f945f14d4739b9fb165b5a945ab7
I20250115 20:50:25.392041 29457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1195eca7127442e7a2c5a9e4df7b679a"
I20250115 20:50:25.394109 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a8f945f14d4739b9fb165b5a945ab7 in term 2.
I20250115 20:50:25.394285 29565 raft_consensus.cc:2743] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 20:50:25.394519 29288 leader_election.cc:304] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43a8f945f14d4739b9fb165b5a945ab7, af9e84ab837a4c4face479825e0bded7; no voters: 1195eca7127442e7a2c5a9e4df7b679a
I20250115 20:50:25.394784 29564 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 FOLLOWER]: Leader election won for term 2
I20250115 20:50:25.395217 29564 raft_consensus.cc:695] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 LEADER]: Becoming Leader. State: Replica: 43a8f945f14d4739b9fb165b5a945ab7, State: Running, Role: LEADER
I20250115 20:50:25.395675 29564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.397145 29566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a8f945f14d4739b9fb165b5a945ab7. Latest consensus state: current_term: 2 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:25.397325 29566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: LEADER
I20250115 20:50:25.397935 29567 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 20:50:25.399664 29545 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:25.400107 29567 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 20:50:25.400707 29545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:25.400975 29567 catalog_manager.cc:1260] Loaded cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:25.401048 29567 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 20:50:25.401023 29545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:25.402287 29545 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 20:50:25.402369 29567 catalog_manager.cc:1505] Loading token signing keys...
I20250115 20:50:25.403240 29567 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Loaded TSK: 0
I20250115 20:50:25.404337 29567 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 20:50:25.404455 29545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 20:50:25.404546 29311 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 LEADER]: Rejecting Update request from peer 1195eca7127442e7a2c5a9e4df7b679a for earlier term 1. Current term is 2. Ops: []
I20250115 20:50:25.404608 29545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250115 20:50:25.404742 29545 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 20:50:25.404805 29545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:25.405575 29569 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:25.406109 29569 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Peer 43a8f945f14d4739b9fb165b5a945ab7 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250115 20:50:25.406258 29572 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 LEADER]: Stepping down as leader of term 1
I20250115 20:50:25.406337 29572 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1195eca7127442e7a2c5a9e4df7b679a, State: Running, Role: LEADER
I20250115 20:50:25.406667 29572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.407196 29572 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 1 FOLLOWER]: Advancing to term 2
I20250115 20:50:25.414299 29502 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 20:50:25.438874 29545 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.193:41569
I20250115 20:50:25.438905 29677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.193:41569 every 8 connection(s)
I20250115 20:50:25.440337 29545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/data/info.pb
I20250115 20:50:25.442492 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29545
I20250115 20:50:25.442734 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/instance
I20250115 20:50:25.451951 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.147.194:0
--local_ip_for_outbound_sockets=127.28.147.194
--webserver_interface=127.28.147.194
--webserver_port=0
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250115 20:50:25.461839 29679 heartbeater.cc:346] Connected to a master server at 127.28.147.254:45409
I20250115 20:50:25.462240 29679 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.463172 29679 heartbeater.cc:509] Master 127.28.147.254:45409 requested a full tablet report, sending...
I20250115 20:50:25.465622 29302 ts_manager.cc:194] Registered new tserver with Master: 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569)
I20250115 20:50:25.467945 29302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.147.193:37343
I20250115 20:50:25.472774 29680 heartbeater.cc:346] Connected to a master server at 127.28.147.253:46053
I20250115 20:50:25.472934 29680 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.473328 29680 heartbeater.cc:509] Master 127.28.147.253:46053 requested a full tablet report, sending...
I20250115 20:50:25.474889 29375 ts_manager.cc:194] Registered new tserver with Master: 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569)
I20250115 20:50:25.481056 29678 heartbeater.cc:346] Connected to a master server at 127.28.147.252:38139
I20250115 20:50:25.481222 29678 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.481489 29678 heartbeater.cc:509] Master 127.28.147.252:38139 requested a full tablet report, sending...
I20250115 20:50:25.483347 29446 ts_manager.cc:194] Registered new tserver with Master: 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569)
W20250115 20:50:25.796769 29688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:25.797071 29688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:25.797183 29688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:25.804947 29688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:25.805164 29688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.147.194
I20250115 20:50:25.814658 29688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.147.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.147.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.147.194
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:25.816217 29688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:25.817939 29688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:25.829823 29694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:25.830505 29695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:25.831158 29697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:25.831948 29688 server_base.cc:1034] running on GCE node
I20250115 20:50:25.832468 29688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:25.833411 29688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:25.834643 29688 hybrid_clock.cc:648] HybridClock initialized: now 1736974225834552 us; error 49 us; skew 500 ppm
I20250115 20:50:25.834942 29688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:25.837250 29688 webserver.cc:458] Webserver started at http://127.28.147.194:45807/ using document root <none> and password file <none>
I20250115 20:50:25.837983 29688 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:25.838110 29688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:25.838413 29688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:25.841090 29688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data/instance:
uuid: "efbc16385f7f43bdad120af5153d3b90"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.841806 29688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal/instance:
uuid: "efbc16385f7f43bdad120af5153d3b90"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.847474 29688 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250115 20:50:25.851043 29703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:25.852425 29688 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250115 20:50:25.852631 29688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal
uuid: "efbc16385f7f43bdad120af5153d3b90"
format_stamp: "Formatted at 2025-01-15 20:50:25 on dist-test-slave-2pn0"
I20250115 20:50:25.852839 29688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:25.881740 29688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:25.882606 29688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:25.882886 29688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:25.883919 29688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 20:50:25.885623 29688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 20:50:25.885718 29688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:25.885847 29688 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 20:50:25.885916 29688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:25.918447 29688 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.194:46775
I20250115 20:50:25.918509 29816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.194:46775 every 8 connection(s)
I20250115 20:50:25.919801 29688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/data/info.pb
I20250115 20:50:25.920847 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29688
I20250115 20:50:25.921044 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal/instance
I20250115 20:50:25.928314 29263 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
/tmp/dist-test-taskLfmlVi/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.147.195:0
--local_ip_for_outbound_sockets=127.28.147.195
--webserver_interface=127.28.147.195
--webserver_port=0
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--builtin_ntp_servers=127.28.147.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250115 20:50:25.936414 29814 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 20:50:25.936764 29814 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.937800 29814 leader_election.cc:290] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053)
I20250115 20:50:25.938690 29311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7" is_pre_election: true
I20250115 20:50:25.939302 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7" is_pre_election: true
I20250115 20:50:25.939565 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1195eca7127442e7a2c5a9e4df7b679a in term 2.
I20250115 20:50:25.940724 29432 leader_election.cc:304] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1195eca7127442e7a2c5a9e4df7b679a, af9e84ab837a4c4face479825e0bded7; no voters: 43a8f945f14d4739b9fb165b5a945ab7
I20250115 20:50:25.941268 29814 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250115 20:50:25.941386 29814 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 20:50:25.941514 29814 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 2 FOLLOWER]: Advancing to term 3
I20250115 20:50:25.945168 29814 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.946424 29311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7"
I20250115 20:50:25.947288 29814 leader_election.cc:290] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 3 election: Requested vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053)
I20250115 20:50:25.947985 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7"
I20250115 20:50:25.948232 29383 raft_consensus.cc:3054] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 2 FOLLOWER]: Advancing to term 3
I20250115 20:50:25.950852 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1195eca7127442e7a2c5a9e4df7b679a in term 3.
I20250115 20:50:25.951136 29818 heartbeater.cc:346] Connected to a master server at 127.28.147.254:45409
I20250115 20:50:25.951512 29818 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.952399 29818 heartbeater.cc:509] Master 127.28.147.254:45409 requested a full tablet report, sending...
I20250115 20:50:25.953397 29432 leader_election.cc:304] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1195eca7127442e7a2c5a9e4df7b679a, af9e84ab837a4c4face479825e0bded7; no voters: 43a8f945f14d4739b9fb165b5a945ab7
I20250115 20:50:25.953790 29814 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 FOLLOWER]: Leader election won for term 3
I20250115 20:50:25.954005 29302 ts_manager.cc:194] Registered new tserver with Master: efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775)
I20250115 20:50:25.955624 29302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.147.194:40933
I20250115 20:50:25.956151 29817 heartbeater.cc:346] Connected to a master server at 127.28.147.252:38139
I20250115 20:50:25.956336 29817 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.957195 29817 heartbeater.cc:509] Master 127.28.147.252:38139 requested a full tablet report, sending...
I20250115 20:50:25.957612 29820 heartbeater.cc:346] Connected to a master server at 127.28.147.253:46053
I20250115 20:50:25.957804 29820 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:25.958709 29820 heartbeater.cc:509] Master 127.28.147.253:46053 requested a full tablet report, sending...
I20250115 20:50:25.961076 29814 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 LEADER]: Becoming Leader. State: Replica: 1195eca7127442e7a2c5a9e4df7b679a, State: Running, Role: LEADER
I20250115 20:50:25.961256 29375 ts_manager.cc:194] Registered new tserver with Master: efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775)
I20250115 20:50:25.961441 29814 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.963183 29446 ts_manager.cc:194] Registered new tserver with Master: efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775)
I20250115 20:50:25.963304 29825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1195eca7127442e7a2c5a9e4df7b679a. Latest consensus state: current_term: 3 leader_uuid: "1195eca7127442e7a2c5a9e4df7b679a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:25.963551 29825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: LEADER
I20250115 20:50:25.964097 29827 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 20:50:25.965668 29827 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 20:50:25.966481 29827 catalog_manager.cc:1260] Loaded cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:25.966553 29827 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 20:50:25.967880 29827 catalog_manager.cc:1505] Loading token signing keys...
I20250115 20:50:25.968704 29827 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Loaded TSK: 0
I20250115 20:50:25.969350 29827 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 20:50:25.971247 29679 heartbeater.cc:501] Master 127.28.147.254:45409 was elected leader, sending a full tablet report...
I20250115 20:50:25.986341 29383 raft_consensus.cc:1235] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 3 FOLLOWER]: Rejecting Update request from peer 43a8f945f14d4739b9fb165b5a945ab7 for earlier term 2. Current term is 3. Ops: []
I20250115 20:50:25.987090 29828 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:25.987797 29828 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 LEADER]: Stepping down as leader of term 2
I20250115 20:50:25.987941 29828 raft_consensus.cc:738] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 43a8f945f14d4739b9fb165b5a945ab7, State: Running, Role: LEADER
I20250115 20:50:25.988260 29828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:25.988823 29828 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 2 FOLLOWER]: Advancing to term 3
W20250115 20:50:26.269302 29823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 20:50:26.269541 29823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 20:50:26.269635 29823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 20:50:26.276597 29823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 20:50:26.276793 29823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.147.195
I20250115 20:50:26.285372 29823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.147.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.147.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.147.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.28.147.254:45409,127.28.147.253:46053,127.28.147.252:38139
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.147.195
--log_dir=/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4195
ASAN enabled
I20250115 20:50:26.286852 29823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 20:50:26.288414 29823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 20:50:26.299453 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:26.300187 29835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 20:50:26.300760 29837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 20:50:26.301443 29823 server_base.cc:1034] running on GCE node
I20250115 20:50:26.301986 29823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 20:50:26.302806 29823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 20:50:26.304023 29823 hybrid_clock.cc:648] HybridClock initialized: now 1736974226303948 us; error 49 us; skew 500 ppm
I20250115 20:50:26.304291 29823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 20:50:26.306392 29823 webserver.cc:458] Webserver started at http://127.28.147.195:45203/ using document root <none> and password file <none>
I20250115 20:50:26.307080 29823 fs_manager.cc:362] Metadata directory not provided
I20250115 20:50:26.307210 29823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 20:50:26.307487 29823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 20:50:26.313232 29823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data/instance:
uuid: "263078466c184592adf0cd2a8d6593d2"
format_stamp: "Formatted at 2025-01-15 20:50:26 on dist-test-slave-2pn0"
I20250115 20:50:26.313943 29823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal/instance:
uuid: "263078466c184592adf0cd2a8d6593d2"
format_stamp: "Formatted at 2025-01-15 20:50:26 on dist-test-slave-2pn0"
I20250115 20:50:26.318696 29823 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250115 20:50:26.321681 29844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:26.322888 29823 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250115 20:50:26.323030 29823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal
uuid: "263078466c184592adf0cd2a8d6593d2"
format_stamp: "Formatted at 2025-01-15 20:50:26 on dist-test-slave-2pn0"
I20250115 20:50:26.323191 29823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 20:50:26.345067 29823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 20:50:26.345829 29823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 20:50:26.346181 29823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 20:50:26.347179 29823 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 20:50:26.348647 29823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 20:50:26.348722 29823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:26.348850 29823 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 20:50:26.348912 29823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 20:50:26.374289 29828 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 20:50:26.374559 29828 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:26.375574 29828 leader_election.cc:290] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053), 1195eca7127442e7a2c5a9e4df7b679a (127.28.147.252:38139)
I20250115 20:50:26.376075 29457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1195eca7127442e7a2c5a9e4df7b679a" is_pre_election: true
I20250115 20:50:26.376163 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7" is_pre_election: true
I20250115 20:50:26.376490 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a8f945f14d4739b9fb165b5a945ab7 in term 3.
I20250115 20:50:26.376989 29288 leader_election.cc:304] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43a8f945f14d4739b9fb165b5a945ab7, af9e84ab837a4c4face479825e0bded7; no voters: 1195eca7127442e7a2c5a9e4df7b679a
I20250115 20:50:26.377386 29828 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250115 20:50:26.377513 29828 raft_consensus.cc:491] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 20:50:26.377664 29828 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 3 FOLLOWER]: Advancing to term 4
I20250115 20:50:26.381469 29828 raft_consensus.cc:513] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:26.382391 29828 leader_election.cc:290] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 4 election: Requested vote from peers af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053), 1195eca7127442e7a2c5a9e4df7b679a (127.28.147.252:38139)
I20250115 20:50:26.383001 29383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7"
I20250115 20:50:26.383241 29383 raft_consensus.cc:3054] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 3 FOLLOWER]: Advancing to term 4
I20250115 20:50:26.383208 29457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "43a8f945f14d4739b9fb165b5a945ab7" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "1195eca7127442e7a2c5a9e4df7b679a"
I20250115 20:50:26.385464 29383 raft_consensus.cc:2463] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a8f945f14d4739b9fb165b5a945ab7 in term 4.
I20250115 20:50:26.386013 29288 leader_election.cc:304] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 43a8f945f14d4739b9fb165b5a945ab7, af9e84ab837a4c4face479825e0bded7; no voters: 1195eca7127442e7a2c5a9e4df7b679a
I20250115 20:50:26.386318 29828 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 4 FOLLOWER]: Leader election won for term 4
I20250115 20:50:26.386533 29828 raft_consensus.cc:695] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 4 LEADER]: Becoming Leader. State: Replica: 43a8f945f14d4739b9fb165b5a945ab7, State: Running, Role: LEADER
I20250115 20:50:26.386817 29828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:26.387923 29829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a8f945f14d4739b9fb165b5a945ab7. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.388233 29829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: LEADER
I20250115 20:50:26.389003 29949 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 20:50:26.391408 29949 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 20:50:26.392062 29957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.28.147.195:39249 every 8 connection(s)
I20250115 20:50:26.392058 29823 rpc_server.cc:307] RPC server started. Bound to: 127.28.147.195:39249
I20250115 20:50:26.392483 29949 catalog_manager.cc:1260] Loaded cluster ID: 7d1265ea851b47f5b52e0bb122a25fed
I20250115 20:50:26.392585 29949 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 20:50:26.393960 29823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/data/info.pb
I20250115 20:50:26.394062 29949 catalog_manager.cc:1505] Loading token signing keys...
I20250115 20:50:26.395211 29949 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Loaded TSK: 0
I20250115 20:50:26.395977 29949 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 20:50:26.403209 29263 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu as pid 29823
I20250115 20:50:26.403375 29263 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal/instance
I20250115 20:50:26.412567 29958 heartbeater.cc:346] Connected to a master server at 127.28.147.252:38139
I20250115 20:50:26.412966 29958 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:26.413957 29958 heartbeater.cc:509] Master 127.28.147.252:38139 requested a full tablet report, sending...
I20250115 20:50:26.414646 29959 heartbeater.cc:346] Connected to a master server at 127.28.147.254:45409
I20250115 20:50:26.414795 29959 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:26.415305 29959 heartbeater.cc:509] Master 127.28.147.254:45409 requested a full tablet report, sending...
I20250115 20:50:26.415416 29446 ts_manager.cc:194] Registered new tserver with Master: 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:26.416241 29302 ts_manager.cc:194] Registered new tserver with Master: 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:26.416625 29960 heartbeater.cc:346] Connected to a master server at 127.28.147.253:46053
I20250115 20:50:26.416762 29960 heartbeater.cc:463] Registering TS with master...
I20250115 20:50:26.416978 29446 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.147.195:53381
I20250115 20:50:26.417246 29960 heartbeater.cc:509] Master 127.28.147.253:46053 requested a full tablet report, sending...
I20250115 20:50:26.417804 29302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.147.195:41421
I20250115 20:50:26.418316 29375 ts_manager.cc:194] Registered new tserver with Master: 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:26.429558 29263 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250115 20:50:26.460198 29818 heartbeater.cc:501] Master 127.28.147.254:45409 was elected leader, sending a full tablet report...
I20250115 20:50:26.459949 29442 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_4dc4f03a55c044618d84b585753a3cf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.460872 29441 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_e598a3eeca4b42a893418f484416b263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.462347 29442 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc4f03a55c044618d84b585753a3cf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.462519 29440 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_e575a6b345e04ac398a61854d6fe12e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.463064 29440 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e575a6b345e04ac398a61854d6fe12e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.463430 29439 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_35426724596b4053b606282a49b46b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.463966 29439 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35426724596b4053b606282a49b46b4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.460050 29443 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_cf97ccb2bdd24a07a16568cd8d1d652e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.464843 29443 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf97ccb2bdd24a07a16568cd8d1d652e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.464035 29438 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_faa825d8fe1e4205a2a6d86b2a48f8a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.467002 29441 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e598a3eeca4b42a893418f484416b263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:26.467917 29438 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faa825d8fe1e4205a2a6d86b2a48f8a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.468747 29446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_32fd53ca865649df8e9749016d40b10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.469221 29446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32fd53ca865649df8e9749016d40b10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.470172 29444 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_3c2c61d4aa8541b99facbfc1840f8385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.470721 29444 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c2c61d4aa8541b99facbfc1840f8385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.469870 29445 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_098be743be114975822d862af07b7181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.474335 29445 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_098be743be114975822d862af07b7181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.475600 29437 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42030:
name: "default.Table_ae5594f7e17f4a20bb5795db903f4dae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.476325 29437 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae5594f7e17f4a20bb5795db903f4dae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.481778 29311 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [term 4 LEADER]: Rejecting Update request from peer 1195eca7127442e7a2c5a9e4df7b679a for earlier term 3. Current term is 4. Ops: []
I20250115 20:50:26.481927 29383 raft_consensus.cc:1235] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 4 FOLLOWER]: Rejecting Update request from peer 1195eca7127442e7a2c5a9e4df7b679a for earlier term 3. Current term is 4. Ops: []
I20250115 20:50:26.482486 29985 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:26.482995 29985 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Peer 43a8f945f14d4739b9fb165b5a945ab7 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250115 20:50:26.483170 29985 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 LEADER]: Stepping down as leader of term 3
I20250115 20:50:26.483281 29985 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1195eca7127442e7a2c5a9e4df7b679a, State: Running, Role: LEADER
I20250115 20:50:26.483233 29991 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:26.483848 29985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:26.484956 29985 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 3 FOLLOWER]: Advancing to term 4
W20250115 20:50:26.485772 29439 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.004s ago.
W20250115 20:50:26.486450 29442 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.005s ago.
I20250115 20:50:26.486759 29678 heartbeater.cc:501] Master 127.28.147.252:38139 was elected leader, sending a full tablet report...
W20250115 20:50:26.487104 29441 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.006s ago.
W20250115 20:50:26.487982 29446 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.007s ago.
W20250115 20:50:26.488188 29966 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.147.252:38139)
W20250115 20:50:26.488950 29444 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.008s ago.
W20250115 20:50:26.489856 29438 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.008s ago.
W20250115 20:50:26.490664 29443 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.009s ago.
W20250115 20:50:26.491479 29445 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.010s ago.
W20250115 20:50:26.492341 29437 catalog_manager.cc:2177] Illegal state: an error occurred while writing to the sys-catalog: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.011s ago.
I20250115 20:50:26.493168 29302 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_35426724596b4053b606282a49b46b4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.494899 29302 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35426724596b4053b606282a49b46b4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.493131 29301 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_4dc4f03a55c044618d84b585753a3cf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.497009 29301 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dc4f03a55c044618d84b585753a3cf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.499241 29297 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_faa825d8fe1e4205a2a6d86b2a48f8a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.499689 29296 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_cf97ccb2bdd24a07a16568cd8d1d652e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.499931 29297 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faa825d8fe1e4205a2a6d86b2a48f8a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:26.500270 29296 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf97ccb2bdd24a07a16568cd8d1d652e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.501477 29295 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_098be743be114975822d862af07b7181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.502017 29295 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_098be743be114975822d862af07b7181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.503065 29294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_ae5594f7e17f4a20bb5795db903f4dae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.503556 29294 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae5594f7e17f4a20bb5795db903f4dae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.504524 29298 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_3c2c61d4aa8541b99facbfc1840f8385"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.505124 29298 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c2c61d4aa8541b99facbfc1840f8385 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.509459 29457 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 4 FOLLOWER]: Refusing update from remote peer 43a8f945f14d4739b9fb165b5a945ab7: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250115 20:50:26.509507 29383 raft_consensus.cc:1270] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [term 4 FOLLOWER]: Refusing update from remote peer 43a8f945f14d4739b9fb165b5a945ab7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20250115 20:50:26.510128 29829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:26.510356 29828 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250115 20:50:26.510560 29828 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [LEADER]: Peer 1195eca7127442e7a2c5a9e4df7b679a log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250115 20:50:26.512270 29457 pending_rounds.cc:77] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Aborting all ops after (but not including) 4
I20250115 20:50:26.512418 29457 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20250115 20:50:26.512544 29457 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250115 20:50:26.512670 29457 pending_rounds.cc:99] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a: Aborting uncommitted WRITE_OP operation due to leader change: 3.6
I20250115 20:50:26.512784 29300 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e598a3eeca4b42a893418f484416b263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.513340 29300 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e598a3eeca4b42a893418f484416b263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:26.513366 29440 catalog_manager.cc:2177] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20250115 20:50:26.516417 29299 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_32fd53ca865649df8e9749016d40b10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.517207 29299 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32fd53ca865649df8e9749016d40b10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.521345 29985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a8f945f14d4739b9fb165b5a945ab7. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.521606 29985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.523149 29293 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e575a6b345e04ac398a61854d6fe12e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.523801 29293 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e575a6b345e04ac398a61854d6fe12e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.527103 29994 mvcc.cc:204] Tried to move back new op lower bound from 7114646431775879168 to 7114646427224784896. Current Snapshot: MvccSnapshot[applied={T|T < 7114646431775879168}]
I20250115 20:50:26.539865 29302 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_35426724596b4053b606282a49b46b4e" } new_table_name: "default.Table_04f15411768240efb1413c3baf65ef19" modify_external_catalogs: true
I20250115 20:50:26.539314 29301 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_4dc4f03a55c044618d84b585753a3cf5" } new_table_name: "default.Table_e3b5a2b22a3f4a768c449c7545648f1a" modify_external_catalogs: true
I20250115 20:50:26.544984 29996 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43a8f945f14d4739b9fb165b5a945ab7. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.545668 29996 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.546320 29829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.546666 29829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: LEADER
I20250115 20:50:26.548249 29988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.548368 29985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.548663 29988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.548709 29985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.592144 30010 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.592545 30010 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.592898 29996 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.593093 29996 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e84ab837a4c4face479825e0bded7 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 20:50:26.606796 29297 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_faa825d8fe1e4205a2a6d86b2a48f8a8" } new_table_name: "default.Table_edb5a8ea4b874042bace585191f29f03" modify_external_catalogs: true
I20250115 20:50:26.608495 29296 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_cf97ccb2bdd24a07a16568cd8d1d652e" } new_table_name: "default.Table_b14574f1380b46709a3e46b58e8d1e87" modify_external_catalogs: true
I20250115 20:50:26.610222 29298 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_32fd53ca865649df8e9749016d40b10f" } modify_external_catalogs: true
I20250115 20:50:26.610472 29298 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_32fd53ca865649df8e9749016d40b10f" } modify_external_catalogs: true
I20250115 20:50:26.612257 29294 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e575a6b345e04ac398a61854d6fe12e6" } modify_external_catalogs: true
I20250115 20:50:26.612493 29294 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e575a6b345e04ac398a61854d6fe12e6" } modify_external_catalogs: true
I20250115 20:50:26.613055 29293 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e598a3eeca4b42a893418f484416b263" } modify_external_catalogs: true
I20250115 20:50:26.613005 29299 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_3c2c61d4aa8541b99facbfc1840f8385" } modify_external_catalogs: true
I20250115 20:50:26.613227 29295 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_098be743be114975822d862af07b7181" } new_table_name: "default.Table_50662e48c81c42ffa80cfc00eebf705a" modify_external_catalogs: true
I20250115 20:50:26.613237 29293 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e598a3eeca4b42a893418f484416b263" } modify_external_catalogs: true
I20250115 20:50:26.615271 29300 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_ae5594f7e17f4a20bb5795db903f4dae" } modify_external_catalogs: true
I20250115 20:50:26.613245 29299 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_3c2c61d4aa8541b99facbfc1840f8385" } modify_external_catalogs: true
I20250115 20:50:26.615509 29300 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_ae5594f7e17f4a20bb5795db903f4dae" } modify_external_catalogs: true
I20250115 20:50:26.622555 29828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "43a8f945f14d4739b9fb165b5a945ab7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } } }
I20250115 20:50:26.622905 29828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7 [sys.catalog]: This master's current role is: LEADER
W20250115 20:50:26.629096 29302 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f175ae17b4e04f7a99b5ce55cdec12bf consensus state has no leader
W20250115 20:50:26.629717 29301 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cfd32f2f4cb245379491ca1ed81f600c consensus state has no leader
W20250115 20:50:26.630116 29297 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16d2fc924b454a438e949b65a1b1575f has no consensus state
W20250115 20:50:26.656262 29296 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24de8c78c961473297a1f3713f438031 has no consensus state
I20250115 20:50:26.656870 29298 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8b18cf8f826e4af18df0bbd321a3f8de
I20250115 20:50:26.657006 29293 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 398508442f1045c6817ae7a28c685cf3
I20250115 20:50:26.657653 29263 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskLfmlVi/build/asan/bin/kudu pid 29413
W20250115 20:50:26.657811 29295 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07b80aa3def740d6b6ba07698a9a6153 has no consensus state
I20250115 20:50:26.658041 29299 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 29d4f3a146254404aad5856cd18cf309
I20250115 20:50:26.658953 29477 generic_service.cc:196] Checking for leaks (request via RPC)
I20250115 20:50:26.661933 29613 tablet_service.cc:1467] Processing CreateTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf (DEFAULT_TABLE table=default.Table_35426724596b4053b606282a49b46b4e [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.663810 29613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f175ae17b4e04f7a99b5ce55cdec12bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.665023 29612 tablet_service.cc:1467] Processing CreateTablet for tablet cfd32f2f4cb245379491ca1ed81f600c (DEFAULT_TABLE table=default.Table_4dc4f03a55c044618d84b585753a3cf5 [id=6053566edd8b46e8879c858544edc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.665580 29612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd32f2f4cb245379491ca1ed81f600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.668660 29294 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 01c232b4a6684e1a83002edc8c80dfb0
I20250115 20:50:26.669099 29300 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 953bfeaeaf544f5687fc9e3d9b1e391d
W20250115 20:50:26.670604 29288 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f175ae17b4e04f7a99b5ce55cdec12bf (table default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f175ae17b4e04f7a99b5ce55cdec12bf consensus state has no leader
I20250115 20:50:26.683254 29610 tablet_service.cc:1467] Processing CreateTablet for tablet 24de8c78c961473297a1f3713f438031 (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.683396 29609 tablet_service.cc:1467] Processing CreateTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.684543 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24de8c78c961473297a1f3713f438031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.685081 29611 tablet_service.cc:1467] Processing CreateTablet for tablet 16d2fc924b454a438e949b65a1b1575f (DEFAULT_TABLE table=default.Table_edb5a8ea4b874042bace585191f29f03 [id=944bf40a4d9241fcb1323e0f749ee4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.685794 29611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d2fc924b454a438e949b65a1b1575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.688663 29750 tablet_service.cc:1467] Processing CreateTablet for tablet cfd32f2f4cb245379491ca1ed81f600c (DEFAULT_TABLE table=default.Table_4dc4f03a55c044618d84b585753a3cf5 [id=6053566edd8b46e8879c858544edc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.690524 29750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd32f2f4cb245379491ca1ed81f600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.693291 29749 tablet_service.cc:1467] Processing CreateTablet for tablet 16d2fc924b454a438e949b65a1b1575f (DEFAULT_TABLE table=default.Table_edb5a8ea4b874042bace585191f29f03 [id=944bf40a4d9241fcb1323e0f749ee4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.693888 29749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d2fc924b454a438e949b65a1b1575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.696959 29748 tablet_service.cc:1467] Processing CreateTablet for tablet 24de8c78c961473297a1f3713f438031 (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.697489 29748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24de8c78c961473297a1f3713f438031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.696789 29892 tablet_service.cc:1467] Processing CreateTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf (DEFAULT_TABLE table=default.Table_35426724596b4053b606282a49b46b4e [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.698406 29747 tablet_service.cc:1467] Processing CreateTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.698630 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f175ae17b4e04f7a99b5ce55cdec12bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.698966 29747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b80aa3def740d6b6ba07698a9a6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.699893 29891 tablet_service.cc:1467] Processing CreateTablet for tablet cfd32f2f4cb245379491ca1ed81f600c (DEFAULT_TABLE table=default.Table_4dc4f03a55c044618d84b585753a3cf5 [id=6053566edd8b46e8879c858544edc22b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.699937 29890 tablet_service.cc:1467] Processing CreateTablet for tablet 16d2fc924b454a438e949b65a1b1575f (DEFAULT_TABLE table=default.Table_edb5a8ea4b874042bace585191f29f03 [id=944bf40a4d9241fcb1323e0f749ee4cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.700644 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.701395 29889 tablet_service.cc:1467] Processing CreateTablet for tablet 24de8c78c961473297a1f3713f438031 (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.703001 29891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfd32f2f4cb245379491ca1ed81f600c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.704917 29889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24de8c78c961473297a1f3713f438031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.705741 29888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b80aa3def740d6b6ba07698a9a6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.706506 29890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d2fc924b454a438e949b65a1b1575f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.707504 29609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07b80aa3def740d6b6ba07698a9a6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.724723 29751 tablet_service.cc:1467] Processing CreateTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf (DEFAULT_TABLE table=default.Table_35426724596b4053b606282a49b46b4e [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.725497 29751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f175ae17b4e04f7a99b5ce55cdec12bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.733201 29300 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_35796d24fb7f41bba0f2046081fbbac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.733961 29300 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35796d24fb7f41bba0f2046081fbbac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.736099 29294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_3ebafaeab31a421199367628770deb0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.736641 29294 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ebafaeab31a421199367628770deb0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.736694 30033 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Bootstrap starting.
I20250115 20:50:26.750113 30033 tablet_bootstrap.cc:654] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.752287 30033 log.cc:826] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Log is configured to *not* fsync() on all Append() calls
I20250115 20:50:26.757330 29610 tablet_service.cc:1467] Processing CreateTablet for tablet 898caf6f94024100b52a3e03009354ff (DEFAULT_TABLE table=default.Table_3ebafaeab31a421199367628770deb0e [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.757412 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 898caf6f94024100b52a3e03009354ff (DEFAULT_TABLE table=default.Table_3ebafaeab31a421199367628770deb0e [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.757870 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898caf6f94024100b52a3e03009354ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.758105 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898caf6f94024100b52a3e03009354ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.761118 29300 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_6919316440d24ac29770402453c77d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.762005 29299 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_7cdaf38b9b7f4279a926cc86bb916cf1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.761118 29294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e023d122549e42a4ba4961294aca3853"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.769110 29294 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e023d122549e42a4ba4961294aca3853 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:26.770704 29299 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cdaf38b9b7f4279a926cc86bb916cf1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:26.771667 29300 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6919316440d24ac29770402453c77d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.762915 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 898caf6f94024100b52a3e03009354ff (DEFAULT_TABLE table=default.Table_3ebafaeab31a421199367628770deb0e [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.775767 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 898caf6f94024100b52a3e03009354ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.778137 30032 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:26.782294 30032 tablet_bootstrap.cc:654] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.784122 30032 log.cc:826] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Log is configured to *not* fsync() on all Append() calls
I20250115 20:50:26.788353 29300 master_service.cc:946] ReplaceTablet: received request to replace tablet 898caf6f94024100b52a3e03009354ff from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:26.792538 29299 master_service.cc:946] ReplaceTablet: received request to replace tablet 07b80aa3def740d6b6ba07698a9a6153 from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:26.795323 29612 tablet_service.cc:1467] Processing CreateTablet for tablet 570c4a9054164d2982ee9cfe6c6589da (DEFAULT_TABLE table=default.Table_35796d24fb7f41bba0f2046081fbbac0 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.795794 29612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c4a9054164d2982ee9cfe6c6589da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.794993 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 570c4a9054164d2982ee9cfe6c6589da (DEFAULT_TABLE table=default.Table_35796d24fb7f41bba0f2046081fbbac0 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.796042 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 570c4a9054164d2982ee9cfe6c6589da (DEFAULT_TABLE table=default.Table_35796d24fb7f41bba0f2046081fbbac0 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.796227 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c4a9054164d2982ee9cfe6c6589da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.796864 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570c4a9054164d2982ee9cfe6c6589da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.800242 29299 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 07b80aa3def740d6b6ba07698a9a6153
I20250115 20:50:26.800949 29299 catalog_manager.cc:6596] ReplaceTablet: tablet 07b80aa3def740d6b6ba07698a9a6153 deleted and replaced by tablet 96b3c24367b84205a25f6ea54c084564
I20250115 20:50:26.801903 30034 tablet_bootstrap.cc:492] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:26.804780 29294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_57093ac9a7d54cc5b1900b59392d1961"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.805426 29294 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57093ac9a7d54cc5b1900b59392d1961 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.805716 29295 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_40945fa2a5354f06b36558cd56f0bbe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.806209 29295 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40945fa2a5354f06b36558cd56f0bbe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.804900 29299 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_cec0c89d33694eee981ea50ba075cf38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.806914 29299 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cec0c89d33694eee981ea50ba075cf38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.806968 29293 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_32a50e1c23cc41ad9c40d61a00f63fcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.807473 29293 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32a50e1c23cc41ad9c40d61a00f63fcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.807746 29612 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b3c24367b84205a25f6ea54c084564) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.807847 29298 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_b949d72bbfc343e9a5aa33d899711ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.808403 29298 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b949d72bbfc343e9a5aa33d899711ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.808729 30033 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: No bootstrap required, opened a new log
I20250115 20:50:26.809224 30033 ts_tablet_manager.cc:1397] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.005s
I20250115 20:50:26.810432 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b3c24367b84205a25f6ea54c084564) from {username='slave'} at 127.0.0.1:37270
W20250115 20:50:26.810770 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 07b80aa3def740d6b6ba07698a9a6153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07b80aa3def740d6b6ba07698a9a6153 already in progress: creating tablet
I20250115 20:50:26.812081 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b3c24367b84205a25f6ea54c084564) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.815093 29300 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 898caf6f94024100b52a3e03009354ff
I20250115 20:50:26.815666 29300 catalog_manager.cc:6596] ReplaceTablet: tablet 898caf6f94024100b52a3e03009354ff deleted and replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3
I20250115 20:50:26.816325 29612 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:51536
W20250115 20:50:26.817034 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 898caf6f94024100b52a3e03009354ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898caf6f94024100b52a3e03009354ff already in progress: creating tablet
I20250115 20:50:26.817360 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.833392 29750 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.834228 29293 master_service.cc:946] ReplaceTablet: received request to replace tablet 570c4a9054164d2982ee9cfe6c6589da from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:26.835995 29298 master_service.cc:946] ReplaceTablet: received request to replace tablet 24de8c78c961473297a1f3713f438031 from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:26.837854 29294 master_service.cc:946] ReplaceTablet: received request to replace tablet f175ae17b4e04f7a99b5ce55cdec12bf from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:26.848675 30034 tablet_bootstrap.cc:654] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.850672 30032 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:26.850978 30032 ts_tablet_manager.cc:1397] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.073s user 0.022s sys 0.031s
W20250115 20:50:26.851058 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 07b80aa3def740d6b6ba07698a9a6153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07b80aa3def740d6b6ba07698a9a6153 already in progress: creating tablet
W20250115 20:50:26.851680 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 898caf6f94024100b52a3e03009354ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898caf6f94024100b52a3e03009354ff already in progress: creating tablet
I20250115 20:50:26.851926 29295 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_edb5a8ea4b874042bace585191f29f03" } new_table_name: "default.Table_61db47e7fb0e4aa5a2528f57b57a74e0" modify_external_catalogs: true
I20250115 20:50:26.852027 29300 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_40945fa2a5354f06b36558cd56f0bbe5" } new_table_name: "default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc" modify_external_catalogs: true
I20250115 20:50:26.853369 29299 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_57093ac9a7d54cc5b1900b59392d1961" } new_table_name: "default.Table_3f15a7a02c404917be2eb509552d3ced" modify_external_catalogs: true
I20250115 20:50:26.856642 30034 log.cc:826] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Log is configured to *not* fsync() on all Append() calls
I20250115 20:50:26.859985 30032 raft_consensus.cc:357] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.860471 30032 raft_consensus.cc:383] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.860596 30032 raft_consensus.cc:738] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.861481 30032 consensus_queue.cc:260] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.863131 29296 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_32a50e1c23cc41ad9c40d61a00f63fcf" } modify_external_catalogs: true
I20250115 20:50:26.863322 29296 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_32a50e1c23cc41ad9c40d61a00f63fcf" } modify_external_catalogs: true
I20250115 20:50:26.863706 29301 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_50662e48c81c42ffa80cfc00eebf705a" } modify_external_catalogs: true
I20250115 20:50:26.863906 29301 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_50662e48c81c42ffa80cfc00eebf705a" } modify_external_catalogs: true
I20250115 20:50:26.868211 29612 tablet_service.cc:1467] Processing CreateTablet for tablet f9f59fb888bd4c90b4f6f89671c71f1c (DEFAULT_TABLE table=default.Table_7cdaf38b9b7f4279a926cc86bb916cf1 [id=67f6e4ad0f04425bb8a1f70fe3d50275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.868748 29612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f59fb888bd4c90b4f6f89671c71f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.868695 30033 raft_consensus.cc:357] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.869118 30033 raft_consensus.cc:383] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.869240 30033 raft_consensus.cc:738] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263078466c184592adf0cd2a8d6593d2, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.869362 29302 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_b14574f1380b46709a3e46b58e8d1e87" } modify_external_catalogs: true
I20250115 20:50:26.869515 29302 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_b14574f1380b46709a3e46b58e8d1e87" } modify_external_catalogs: true
I20250115 20:50:26.870154 29297 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_3ebafaeab31a421199367628770deb0e" } new_table_name: "default.Table_1869e8379d444f6b9868883f319ef48f" modify_external_catalogs: true
I20250115 20:50:26.871780 30032 ts_tablet_manager.cc:1428] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20250115 20:50:26.872833 30033 consensus_queue.cc:260] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
W20250115 20:50:26.873708 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 07b80aa3def740d6b6ba07698a9a6153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07b80aa3def740d6b6ba07698a9a6153 already in progress: creating tablet
W20250115 20:50:26.873979 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 898caf6f94024100b52a3e03009354ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898caf6f94024100b52a3e03009354ff already in progress: creating tablet
I20250115 20:50:26.873979 29892 tablet_service.cc:1467] Processing CreateTablet for tablet f9f59fb888bd4c90b4f6f89671c71f1c (DEFAULT_TABLE table=default.Table_7cdaf38b9b7f4279a926cc86bb916cf1 [id=67f6e4ad0f04425bb8a1f70fe3d50275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.874341 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f59fb888bd4c90b4f6f89671c71f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.874393 29746 tablet_service.cc:1467] Processing CreateTablet for tablet f9f59fb888bd4c90b4f6f89671c71f1c (DEFAULT_TABLE table=default.Table_7cdaf38b9b7f4279a926cc86bb916cf1 [id=67f6e4ad0f04425bb8a1f70fe3d50275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.874819 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9f59fb888bd4c90b4f6f89671c71f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.879395 30032 tablet_bootstrap.cc:492] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:26.881006 30032 tablet_bootstrap.cc:654] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
W20250115 20:50:26.883699 29300 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80b8d0224d7443859c4dbc7dc8774863 has no consensus state
W20250115 20:50:26.885728 29299 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c76efc8ea844a4e975bc064e7efb402 has no consensus state
I20250115 20:50:26.885964 29296 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet e59cbdf415924330964ca78ca8a9dcf9
I20250115 20:50:26.886148 29299 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_35796d24fb7f41bba0f2046081fbbac0" } new_table_name: "default.Table_92dfea87af06467d9ff104073cf60513" modify_external_catalogs: true
I20250115 20:50:26.886335 29296 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_6919316440d24ac29770402453c77d2a" } modify_external_catalogs: true
I20250115 20:50:26.886559 29296 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_6919316440d24ac29770402453c77d2a" } modify_external_catalogs: true
I20250115 20:50:26.888465 29293 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 570c4a9054164d2982ee9cfe6c6589da
I20250115 20:50:26.889086 29293 catalog_manager.cc:6596] ReplaceTablet: tablet 570c4a9054164d2982ee9cfe6c6589da deleted and replaced by tablet 2c9bd162ce374885b0dcc8764d4752c3
I20250115 20:50:26.889410 29293 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_04f15411768240efb1413c3baf65ef19" } modify_external_catalogs: true
I20250115 20:50:26.889580 29293 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_04f15411768240efb1413c3baf65ef19" } modify_external_catalogs: true
I20250115 20:50:26.890089 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 570c4a9054164d2982ee9cfe6c6589da with delete_type TABLET_DATA_DELETED (replaced by tablet 2c9bd162ce374885b0dcc8764d4752c3) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.890308 29298 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 24de8c78c961473297a1f3713f438031
I20250115 20:50:26.890784 29298 catalog_manager.cc:6596] ReplaceTablet: tablet 24de8c78c961473297a1f3713f438031 deleted and replaced by tablet 2f61b837a64942d381647bf1d62d86bf
I20250115 20:50:26.891121 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet 570c4a9054164d2982ee9cfe6c6589da with delete_type TABLET_DATA_DELETED (replaced by tablet 2c9bd162ce374885b0dcc8764d4752c3) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.891234 29750 tablet_service.cc:1514] Processing DeleteTablet for tablet 24de8c78c961473297a1f3713f438031 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61b837a64942d381647bf1d62d86bf) from {username='slave'} at 127.0.0.1:49312
W20250115 20:50:26.891782 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 570c4a9054164d2982ee9cfe6c6589da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c4a9054164d2982ee9cfe6c6589da already in progress: creating tablet
W20250115 20:50:26.892058 29295 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16d2fc924b454a438e949b65a1b1575f consensus state has no leader
I20250115 20:50:26.892045 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 24de8c78c961473297a1f3713f438031 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61b837a64942d381647bf1d62d86bf) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.892587 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 24de8c78c961473297a1f3713f438031 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f61b837a64942d381647bf1d62d86bf) from {username='slave'} at 127.0.0.1:51536
W20250115 20:50:26.893404 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 24de8c78c961473297a1f3713f438031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24de8c78c961473297a1f3713f438031 already in progress: creating tablet
W20250115 20:50:26.894125 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 24de8c78c961473297a1f3713f438031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24de8c78c961473297a1f3713f438031 already in progress: creating tablet
W20250115 20:50:26.894444 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 570c4a9054164d2982ee9cfe6c6589da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c4a9054164d2982ee9cfe6c6589da already in progress: creating tablet
I20250115 20:50:26.894500 29613 tablet_service.cc:1467] Processing CreateTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f (DEFAULT_TABLE table=default.Table_e023d122549e42a4ba4961294aca3853 [id=0f48512293574a0980c88eb48ea7b886]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 20:50:26.894670 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 24de8c78c961473297a1f3713f438031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24de8c78c961473297a1f3713f438031 already in progress: creating tablet
I20250115 20:50:26.895020 29613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f481c13ed83e4eb0b42523c6c87dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.896122 29892 tablet_service.cc:1467] Processing CreateTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f (DEFAULT_TABLE table=default.Table_e023d122549e42a4ba4961294aca3853 [id=0f48512293574a0980c88eb48ea7b886]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.896490 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f481c13ed83e4eb0b42523c6c87dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.900954 29750 tablet_service.cc:1467] Processing CreateTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f (DEFAULT_TABLE table=default.Table_e023d122549e42a4ba4961294aca3853 [id=0f48512293574a0980c88eb48ea7b886]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.901321 29750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f481c13ed83e4eb0b42523c6c87dc49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.901423 29889 tablet_service.cc:1467] Processing CreateTablet for tablet b9a3282b69534422a7c462f32fae13ce (DEFAULT_TABLE table=default.Table_6919316440d24ac29770402453c77d2a [id=a71f0e2f88204785827ead583877f0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.901759 29889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a3282b69534422a7c462f32fae13ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.902681 29958 heartbeater.cc:501] Master 127.28.147.252:38139 was elected leader, sending a full tablet report...
I20250115 20:50:26.904560 29959 heartbeater.cc:501] Master 127.28.147.254:45409 was elected leader, sending a full tablet report...
I20250115 20:50:26.905575 29611 tablet_service.cc:1467] Processing CreateTablet for tablet b9a3282b69534422a7c462f32fae13ce (DEFAULT_TABLE table=default.Table_6919316440d24ac29770402453c77d2a [id=a71f0e2f88204785827ead583877f0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.906037 29611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a3282b69534422a7c462f32fae13ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.906922 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 96b3c24367b84205a25f6ea54c084564 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.907249 29888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b3c24367b84205a25f6ea54c084564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.908818 29609 tablet_service.cc:1467] Processing CreateTablet for tablet 96b3c24367b84205a25f6ea54c084564 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.909237 29609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b3c24367b84205a25f6ea54c084564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.911530 29746 tablet_service.cc:1467] Processing CreateTablet for tablet b9a3282b69534422a7c462f32fae13ce (DEFAULT_TABLE table=default.Table_6919316440d24ac29770402453c77d2a [id=a71f0e2f88204785827ead583877f0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.911859 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9a3282b69534422a7c462f32fae13ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.918484 29748 tablet_service.cc:1467] Processing CreateTablet for tablet 96b3c24367b84205a25f6ea54c084564 (DEFAULT_TABLE table=default.Table_50662e48c81c42ffa80cfc00eebf705a [id=4f644481442b4c8db08935218cce0ef3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:26.918826 29748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96b3c24367b84205a25f6ea54c084564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:26.931582 29612 tablet_service.cc:1514] Processing DeleteTablet for tablet 570c4a9054164d2982ee9cfe6c6589da with delete_type TABLET_DATA_DELETED (replaced by tablet 2c9bd162ce374885b0dcc8764d4752c3) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.932278 29294 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet f175ae17b4e04f7a99b5ce55cdec12bf
W20250115 20:50:26.932402 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 570c4a9054164d2982ee9cfe6c6589da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c4a9054164d2982ee9cfe6c6589da already in progress: creating tablet
I20250115 20:50:26.932790 29294 catalog_manager.cc:6596] ReplaceTablet: tablet f175ae17b4e04f7a99b5ce55cdec12bf deleted and replaced by tablet db481c53f6d3402fb65a20c172db6f85
I20250115 20:50:26.934010 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf with delete_type TABLET_DATA_DELETED (replaced by tablet db481c53f6d3402fb65a20c172db6f85) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.934116 29296 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet b9a3282b69534422a7c462f32fae13ce
W20250115 20:50:26.935606 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet f175ae17b4e04f7a99b5ce55cdec12bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f175ae17b4e04f7a99b5ce55cdec12bf already in progress: creating tablet
I20250115 20:50:26.936251 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a3282b69534422a7c462f32fae13ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:37270
W20250115 20:50:26.937054 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet b9a3282b69534422a7c462f32fae13ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9a3282b69534422a7c462f32fae13ce already in progress: creating tablet
I20250115 20:50:26.938215 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf with delete_type TABLET_DATA_DELETED (replaced by tablet db481c53f6d3402fb65a20c172db6f85) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.939267 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a3282b69534422a7c462f32fae13ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.940703 29609 tablet_service.cc:1514] Processing DeleteTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf with delete_type TABLET_DATA_DELETED (replaced by tablet db481c53f6d3402fb65a20c172db6f85) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.941099 30037 tablet_replica.cc:331] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: stopping tablet replica
W20250115 20:50:26.941097 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet f175ae17b4e04f7a99b5ce55cdec12bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f175ae17b4e04f7a99b5ce55cdec12bf already in progress: creating tablet
W20250115 20:50:26.941419 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet b9a3282b69534422a7c462f32fae13ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9a3282b69534422a7c462f32fae13ce already in progress: creating tablet
I20250115 20:50:26.941704 30037 raft_consensus.cc:2238] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:26.942180 30037 raft_consensus.cc:2267] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:26.942868 29611 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a3282b69534422a7c462f32fae13ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.944146 30037 ts_tablet_manager.cc:1905] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:26.945875 29295 catalog_manager.cc:5618] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249) does not have the latest schema for tablet 16d2fc924b454a438e949b65a1b1575f (table default.Table_61db47e7fb0e4aa5a2528f57b57a74e0 [id=944bf40a4d9241fcb1323e0f749ee4cb]). Expected version 2 got 0
I20250115 20:50:26.946173 30034 tablet_bootstrap.cc:492] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:26.946233 29295 catalog_manager.cc:5618] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249) does not have the latest schema for tablet cfd32f2f4cb245379491ca1ed81f600c (table default.Table_e3b5a2b22a3f4a768c449c7545648f1a [id=6053566edd8b46e8879c858544edc22b]). Expected version 1 got 0
W20250115 20:50:26.946539 29299 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c9bd162ce374885b0dcc8764d4752c3 has no consensus state
I20250115 20:50:26.946707 30034 ts_tablet_manager.cc:1397] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.155s user 0.007s sys 0.006s
I20250115 20:50:26.947345 29301 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 96b3c24367b84205a25f6ea54c084564
I20250115 20:50:26.948464 29302 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2f61b837a64942d381647bf1d62d86bf
W20250115 20:50:26.948797 29297 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8db28a3ed90248b69b7e4934d9e124d3 has no consensus state
I20250115 20:50:26.948835 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b3c24367b84205a25f6ea54c084564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:37270
W20250115 20:50:26.949640 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 96b3c24367b84205a25f6ea54c084564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96b3c24367b84205a25f6ea54c084564 already in progress: creating tablet
I20250115 20:50:26.950353 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b3c24367b84205a25f6ea54c084564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.950703 30037 ts_tablet_manager.cc:1918] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:26.950862 30037 log.cc:1198] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/wals/f175ae17b4e04f7a99b5ce55cdec12bf
I20250115 20:50:26.951300 30037 ts_tablet_manager.cc:1939] T f175ae17b4e04f7a99b5ce55cdec12bf P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting consensus metadata
I20250115 20:50:26.953009 29286 catalog_manager.cc:4872] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): tablet f175ae17b4e04f7a99b5ce55cdec12bf (table default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f]) successfully deleted
I20250115 20:50:26.953626 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet 96b3c24367b84205a25f6ea54c084564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:49312
W20250115 20:50:26.953696 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet b9a3282b69534422a7c462f32fae13ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9a3282b69534422a7c462f32fae13ce already in progress: creating tablet
I20250115 20:50:26.954093 29293 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet db481c53f6d3402fb65a20c172db6f85
W20250115 20:50:26.954177 29288 catalog_manager.cc:4839] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 96b3c24367b84205a25f6ea54c084564 because the tablet was not found. No further retry: Not found: Tablet not found: 96b3c24367b84205a25f6ea54c084564
W20250115 20:50:26.954236 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 96b3c24367b84205a25f6ea54c084564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96b3c24367b84205a25f6ea54c084564 already in progress: creating tablet
I20250115 20:50:26.954620 29302 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_6509ddc7e973469799588a103285c235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.955206 29302 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6509ddc7e973469799588a103285c235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.955610 30032 tablet_bootstrap.cc:492] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:26.955737 30032 ts_tablet_manager.cc:1397] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.050s
I20250115 20:50:26.956492 30032 raft_consensus.cc:357] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.956722 30032 raft_consensus.cc:383] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.956816 30032 raft_consensus.cc:738] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.957063 30032 consensus_queue.cc:260] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.957243 29297 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_a749ae25fbf344fbba58ed4ea747addf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.957700 30032 ts_tablet_manager.cc:1428] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250115 20:50:26.957700 29297 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a749ae25fbf344fbba58ed4ea747addf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.957988 30032 tablet_bootstrap.cc:492] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:26.959733 30032 tablet_bootstrap.cc:654] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.961464 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:51536
W20250115 20:50:26.963603 29286 catalog_manager.cc:4839] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet f175ae17b4e04f7a99b5ce55cdec12bf because the tablet was not found. No further retry: Not found: Tablet not found: f175ae17b4e04f7a99b5ce55cdec12bf
I20250115 20:50:26.966295 30034 raft_consensus.cc:357] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.966776 30034 raft_consensus.cc:383] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.966886 30034 raft_consensus.cc:738] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.967887 29817 heartbeater.cc:501] Master 127.28.147.252:38139 was elected leader, sending a full tablet report...
I20250115 20:50:26.967857 30034 consensus_queue.cc:260] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.971931 30034 ts_tablet_manager.cc:1428] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.025s user 0.004s sys 0.009s
I20250115 20:50:26.972291 30034 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:26.972281 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 24de8c78c961473297a1f3713f438031 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:26.972802 30037 tablet_replica.cc:331] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: stopping tablet replica
I20250115 20:50:26.973065 30037 raft_consensus.cc:2238] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:26.973315 30037 raft_consensus.cc:2267] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:26.973961 30032 tablet_bootstrap.cc:492] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:26.974138 30032 ts_tablet_manager.cc:1397] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250115 20:50:26.974188 30034 tablet_bootstrap.cc:654] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.975153 30032 raft_consensus.cc:357] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.975327 30032 raft_consensus.cc:383] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.975395 30032 raft_consensus.cc:738] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.975697 30032 consensus_queue.cc:260] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.976217 30032 ts_tablet_manager.cc:1428] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 20:50:26.976243 30034 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:26.976454 30034 ts_tablet_manager.cc:1397] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250115 20:50:26.977002 30037 ts_tablet_manager.cc:1905] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:26.977492 30034 raft_consensus.cc:357] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.977782 30034 raft_consensus.cc:383] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:26.977914 30034 raft_consensus.cc:738] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:26.977914 30041 raft_consensus.cc:491] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 20:50:26.978147 30041 raft_consensus.cc:513] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.978230 30034 consensus_queue.cc:260] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:26.980027 30034 ts_tablet_manager.cc:1428] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 20:50:26.980336 30041 leader_election.cc:290] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775), 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:26.980443 30034 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:26.981709 29748 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:26.982033 30038 tablet_replica.cc:331] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: stopping tablet replica
I20250115 20:50:26.982498 30038 raft_consensus.cc:2238] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:26.982889 30038 raft_consensus.cc:2267] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:26.983413 30037 ts_tablet_manager.cc:1918] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:26.983551 30037 log.cc:1198] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/wals/24de8c78c961473297a1f3713f438031
I20250115 20:50:26.983964 30037 ts_tablet_manager.cc:1939] T 24de8c78c961473297a1f3713f438031 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting consensus metadata
I20250115 20:50:26.984475 30032 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:26.984855 30038 ts_tablet_manager.cc:1905] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:26.985157 29297 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_ed3830fa6620490e8ef9969895bccd99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:26.985846 29297 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed3830fa6620490e8ef9969895bccd99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.986265 30032 tablet_bootstrap.cc:654] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:26.959211 30033 ts_tablet_manager.cc:1428] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Time spent starting tablet: real 0.150s user 0.020s sys 0.003s
I20250115 20:50:26.964691 29890 tablet_service.cc:1514] Processing DeleteTablet for tablet f175ae17b4e04f7a99b5ce55cdec12bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.964764 29891 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b3c24367b84205a25f6ea54c084564) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.989778 30039 tablet_replica.cc:331] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: stopping tablet replica
I20250115 20:50:26.990022 29295 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_028a64030c8141c89ccd3e63992c45ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.990193 30039 raft_consensus.cc:2238] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:26.990496 30039 raft_consensus.cc:2267] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 20:50:26.990540 29295 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_028a64030c8141c89ccd3e63992c45ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.964787 29888 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.991461 29302 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_cf9040b1c4a34c71aee275e501cd7a90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:26.992129 30039 ts_tablet_manager.cc:1905] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 20:50:26.992681 29302 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf9040b1c4a34c71aee275e501cd7a90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:26.969897 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet 24de8c78c961473297a1f3713f438031 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:26 UTC) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:26.977756 29892 tablet_service.cc:1514] Processing DeleteTablet for tablet 570c4a9054164d2982ee9cfe6c6589da with delete_type TABLET_DATA_DELETED (replaced by tablet 2c9bd162ce374885b0dcc8764d4752c3) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:27.005821 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 898caf6f94024100b52a3e03009354ff with delete_type TABLET_DATA_DELETED (replaced by tablet 8db28a3ed90248b69b7e4934d9e124d3) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:27.006166 30037 tablet_replica.cc:331] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: stopping tablet replica
I20250115 20:50:27.006367 30037 raft_consensus.cc:2238] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:27.006508 29300 master_service.cc:946] ReplaceTablet: received request to replace tablet 570c4a9054164d2982ee9cfe6c6589da from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:27.006562 30037 raft_consensus.cc:2267] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:27.007193 29286 catalog_manager.cc:4872] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): tablet 24de8c78c961473297a1f3713f438031 (table default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]) successfully deleted
I20250115 20:50:27.007387 30037 ts_tablet_manager.cc:1905] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:27.009181 30039 ts_tablet_manager.cc:1918] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.009354 30039 log.cc:1198] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal/wals/f175ae17b4e04f7a99b5ce55cdec12bf
I20250115 20:50:27.009840 30033 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: Bootstrap starting.
I20250115 20:50:27.011626 30033 tablet_bootstrap.cc:654] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.012708 30039 ts_tablet_manager.cc:1939] T f175ae17b4e04f7a99b5ce55cdec12bf P 263078466c184592adf0cd2a8d6593d2: Deleting consensus metadata
I20250115 20:50:27.013888 30037 ts_tablet_manager.cc:1918] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.014027 30037 log.cc:1198] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/wals/898caf6f94024100b52a3e03009354ff
I20250115 20:50:27.014438 30037 ts_tablet_manager.cc:1939] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting consensus metadata
I20250115 20:50:27.017656 29286 catalog_manager.cc:4872] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): tablet 898caf6f94024100b52a3e03009354ff (table default.Table_1869e8379d444f6b9868883f319ef48f [id=944ae62c7cca451687ee750d7b474fe9]) successfully deleted
I20250115 20:50:27.017916 29286 catalog_manager.cc:4872] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): tablet f175ae17b4e04f7a99b5ce55cdec12bf (table default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f]) successfully deleted
W20250115 20:50:27.018127 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 07b80aa3def740d6b6ba07698a9a6153 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07b80aa3def740d6b6ba07698a9a6153 already in progress: creating tablet
W20250115 20:50:27.018314 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 898caf6f94024100b52a3e03009354ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 898caf6f94024100b52a3e03009354ff already in progress: creating tablet
W20250115 20:50:27.018503 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 24de8c78c961473297a1f3713f438031 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24de8c78c961473297a1f3713f438031 already in progress: creating tablet
W20250115 20:50:27.018695 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 570c4a9054164d2982ee9cfe6c6589da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 570c4a9054164d2982ee9cfe6c6589da already in progress: creating tablet
I20250115 20:50:27.018599 29912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "898caf6f94024100b52a3e03009354ff" candidate_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263078466c184592adf0cd2a8d6593d2" is_pre_election: true
I20250115 20:50:27.020202 29891 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61b837a64942d381647bf1d62d86bf (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.020465 29885 tablet_service.cc:1467] Processing CreateTablet for tablet a66dfb75e5df40a4b44950456464ffaf (DEFAULT_TABLE table=default.Table_b949d72bbfc343e9a5aa33d899711ea3 [id=31d0dde6855140e3a26915f49cfb4bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.020787 29888 tablet_service.cc:1467] Processing CreateTablet for tablet 0c76efc8ea844a4e975bc064e7efb402 (DEFAULT_TABLE table=default.Table_3f15a7a02c404917be2eb509552d3ced [id=566c5cd82cf343e6a977ef6026aa167e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.021095 29885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66dfb75e5df40a4b44950456464ffaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.021348 29890 tablet_service.cc:1467] Processing CreateTablet for tablet db481c53f6d3402fb65a20c172db6f85 (DEFAULT_TABLE table=default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.021682 29890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db481c53f6d3402fb65a20c172db6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.024150 29887 tablet_service.cc:1467] Processing CreateTablet for tablet 8db28a3ed90248b69b7e4934d9e124d3 (DEFAULT_TABLE table=default.Table_1869e8379d444f6b9868883f319ef48f [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.024674 29886 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9bd162ce374885b0dcc8764d4752c3 (DEFAULT_TABLE table=default.Table_92dfea87af06467d9ff104073cf60513 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.024806 29891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f61b837a64942d381647bf1d62d86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.025239 29887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db28a3ed90248b69b7e4934d9e124d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.026212 29888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c76efc8ea844a4e975bc064e7efb402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.028196 29886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9bd162ce374885b0dcc8764d4752c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.033998 29300 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e80b327eb46e4273b48b6207e0806542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:27.034852 29300 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e80b327eb46e4273b48b6207e0806542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.035970 29293 master_service.cc:946] ReplaceTablet: received request to replace tablet 570c4a9054164d2982ee9cfe6c6589da from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:27.038154 29299 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e7d519a9085e41eba7ed21d614a9ee86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:27.038846 29299 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d519a9085e41eba7ed21d614a9ee86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.039070 29293 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_21516e6dde08453997dbc7d7d7de4c2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:27.039508 29293 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21516e6dde08453997dbc7d7d7de4c2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.040143 29889 tablet_service.cc:1467] Processing CreateTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 (DEFAULT_TABLE table=default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc [id=be06b30297e346faa975743acf955d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.041437 29301 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_e2e1c0dd8e2646cfa6fab8f9fbc1bc15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:27.042085 29301 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2e1c0dd8e2646cfa6fab8f9fbc1bc15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.042383 29892 tablet_service.cc:1467] Processing CreateTablet for tablet a57d470eb9be46d9a14f582b67e2e829 (DEFAULT_TABLE table=default.Table_cec0c89d33694eee981ea50ba075cf38 [id=634b27d151d54d64aed37288141aae6e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 20:50:27.043581 29288 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.28.147.253:46053, user_credentials={real_user=slave}} blocked reactor thread for 44506.2us
I20250115 20:50:27.043753 29296 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_3815a22e9f88477a821e43581926909b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 20:50:27.044207 29296 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3815a22e9f88477a821e43581926909b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.044566 29748 tablet_service.cc:1467] Processing CreateTablet for tablet a57d470eb9be46d9a14f582b67e2e829 (DEFAULT_TABLE table=default.Table_cec0c89d33694eee981ea50ba075cf38 [id=634b27d151d54d64aed37288141aae6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.045034 29748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57d470eb9be46d9a14f582b67e2e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.049468 29750 tablet_service.cc:1467] Processing CreateTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 (DEFAULT_TABLE table=default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc [id=be06b30297e346faa975743acf955d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.049674 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 0c76efc8ea844a4e975bc064e7efb402 (DEFAULT_TABLE table=default.Table_3f15a7a02c404917be2eb509552d3ced [id=566c5cd82cf343e6a977ef6026aa167e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.049952 29750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b8d0224d7443859c4dbc7dc8774863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.050438 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c76efc8ea844a4e975bc064e7efb402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.050977 29749 tablet_service.cc:1467] Processing CreateTablet for tablet 8db28a3ed90248b69b7e4934d9e124d3 (DEFAULT_TABLE table=default.Table_1869e8379d444f6b9868883f319ef48f [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.051501 29749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db28a3ed90248b69b7e4934d9e124d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.052827 29751 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61b837a64942d381647bf1d62d86bf (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.053192 29751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f61b837a64942d381647bf1d62d86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.056865 29747 tablet_service.cc:1467] Processing CreateTablet for tablet db481c53f6d3402fb65a20c172db6f85 (DEFAULT_TABLE table=default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.057287 29747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db481c53f6d3402fb65a20c172db6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.057344 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9bd162ce374885b0dcc8764d4752c3 (DEFAULT_TABLE table=default.Table_92dfea87af06467d9ff104073cf60513 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.057700 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9bd162ce374885b0dcc8764d4752c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.058821 29294 catalog_manager.cc:5618] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775) does not have the latest schema for tablet 16d2fc924b454a438e949b65a1b1575f (table default.Table_61db47e7fb0e4aa5a2528f57b57a74e0 [id=944bf40a4d9241fcb1323e0f749ee4cb]). Expected version 2 got 0
I20250115 20:50:27.060930 29745 tablet_service.cc:1467] Processing CreateTablet for tablet a66dfb75e5df40a4b44950456464ffaf (DEFAULT_TABLE table=default.Table_b949d72bbfc343e9a5aa33d899711ea3 [id=31d0dde6855140e3a26915f49cfb4bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.061520 29745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66dfb75e5df40a4b44950456464ffaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.063604 30034 tablet_bootstrap.cc:654] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.065062 29613 tablet_service.cc:1467] Processing CreateTablet for tablet a57d470eb9be46d9a14f582b67e2e829 (DEFAULT_TABLE table=default.Table_cec0c89d33694eee981ea50ba075cf38 [id=634b27d151d54d64aed37288141aae6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.065165 29611 tablet_service.cc:1467] Processing CreateTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 (DEFAULT_TABLE table=default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc [id=be06b30297e346faa975743acf955d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.065639 29613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57d470eb9be46d9a14f582b67e2e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.065582 29609 tablet_service.cc:1467] Processing CreateTablet for tablet 0c76efc8ea844a4e975bc064e7efb402 (DEFAULT_TABLE table=default.Table_3f15a7a02c404917be2eb509552d3ced [id=566c5cd82cf343e6a977ef6026aa167e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.065975 29612 tablet_service.cc:1467] Processing CreateTablet for tablet 2f61b837a64942d381647bf1d62d86bf (DEFAULT_TABLE table=default.Table_b14574f1380b46709a3e46b58e8d1e87 [id=6406fe0b2ba641e5be90b24d5ae3773f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.066002 29610 tablet_service.cc:1467] Processing CreateTablet for tablet db481c53f6d3402fb65a20c172db6f85 (DEFAULT_TABLE table=default.Table_04f15411768240efb1413c3baf65ef19 [id=b618b2d346934bd39d2ad4656f32463f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.066393 30038 ts_tablet_manager.cc:1918] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.066570 30038 log.cc:1198] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal/wals/898caf6f94024100b52a3e03009354ff
I20250115 20:50:27.066560 29608 tablet_service.cc:1467] Processing CreateTablet for tablet 8db28a3ed90248b69b7e4934d9e124d3 (DEFAULT_TABLE table=default.Table_1869e8379d444f6b9868883f319ef48f [id=944ae62c7cca451687ee750d7b474fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.068212 29607 tablet_service.cc:1467] Processing CreateTablet for tablet 2c9bd162ce374885b0dcc8764d4752c3 (DEFAULT_TABLE table=default.Table_92dfea87af06467d9ff104073cf60513 [id=202a1f8396e74bcb96019982ff3497fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.069243 29609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c76efc8ea844a4e975bc064e7efb402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.072748 29612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f61b837a64942d381647bf1d62d86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.073181 30038 ts_tablet_manager.cc:1939] T 898caf6f94024100b52a3e03009354ff P efbc16385f7f43bdad120af5153d3b90: Deleting consensus metadata
I20250115 20:50:27.076627 29606 tablet_service.cc:1467] Processing CreateTablet for tablet a66dfb75e5df40a4b44950456464ffaf (DEFAULT_TABLE table=default.Table_b949d72bbfc343e9a5aa33d899711ea3 [id=31d0dde6855140e3a26915f49cfb4bad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.077142 29606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a66dfb75e5df40a4b44950456464ffaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.080590 30034 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.080766 30034 ts_tablet_manager.cc:1397] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.001s
I20250115 20:50:27.080899 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db481c53f6d3402fb65a20c172db6f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.081585 30034 raft_consensus.cc:357] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.081804 30034 raft_consensus.cc:383] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.081888 30034 raft_consensus.cc:738] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.082149 30034 consensus_queue.cc:260] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.083262 30032 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.083418 30032 ts_tablet_manager.cc:1397] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.034s
I20250115 20:50:27.084270 30032 raft_consensus.cc:357] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.084494 30032 raft_consensus.cc:383] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.084620 30032 raft_consensus.cc:738] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.084872 30032 consensus_queue.cc:260] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.085143 29608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8db28a3ed90248b69b7e4934d9e124d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.085465 30032 ts_tablet_manager.cc:1428] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250115 20:50:27.085790 30032 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
W20250115 20:50:27.087213 29560 leader_election.cc:343] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): Illegal state: must be running to vote when last-logged opid is not known
I20250115 20:50:27.087720 30032 tablet_bootstrap.cc:654] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.087888 29288 catalog_manager.cc:4872] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): tablet 898caf6f94024100b52a3e03009354ff (table default.Table_1869e8379d444f6b9868883f319ef48f [id=944ae62c7cca451687ee750d7b474fe9]) successfully deleted
I20250115 20:50:27.088779 29607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9bd162ce374885b0dcc8764d4752c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.093155 29889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b8d0224d7443859c4dbc7dc8774863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.093966 29611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80b8d0224d7443859c4dbc7dc8774863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.097348 29295 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e023d122549e42a4ba4961294aca3853" } new_table_name: "default.Table_c6ab6e8deb8549d18b9933ee083505ec" modify_external_catalogs: true
I20250115 20:50:27.098353 29892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a57d470eb9be46d9a14f582b67e2e829. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.098580 29302 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc" } modify_external_catalogs: true
I20250115 20:50:27.098774 29302 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_8a07b4b6fc84494ca1cb46ba0361f0bc" } modify_external_catalogs: true
I20250115 20:50:27.103924 29294 master_service.cc:946] ReplaceTablet: received request to replace tablet 2f61b837a64942d381647bf1d62d86bf from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:27.104200 29298 master_service.cc:946] ReplaceTablet: received request to replace tablet f481c13ed83e4eb0b42523c6c87dc49f from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:27.106622 29300 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_7cdaf38b9b7f4279a926cc86bb916cf1" } new_table_name: "default.Table_1f38bf9b3a2540b89cfcbf6a05791b79" modify_external_catalogs: true
I20250115 20:50:27.106802 29297 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_cec0c89d33694eee981ea50ba075cf38" } new_table_name: "default.Table_f8d0ce283a474cbcabadf92155e49e6f" modify_external_catalogs: true
I20250115 20:50:27.119414 30034 ts_tablet_manager.cc:1428] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
W20250115 20:50:27.119993 29301 catalog_manager.cc:5359] Ignoring report from unknown tablet 07b80aa3def740d6b6ba07698a9a6153
I20250115 20:50:27.121332 30034 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:27.122982 30034 tablet_bootstrap.cc:654] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
W20250115 20:50:27.129242 29562 leader_election.cc:343] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): Not found: Tablet not found: 898caf6f94024100b52a3e03009354ff
I20250115 20:50:27.129455 29562 leader_election.cc:304] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ba117825c9e4c9f9fa8eb9fb9a4efad; no voters: 263078466c184592adf0cd2a8d6593d2, efbc16385f7f43bdad120af5153d3b90
W20250115 20:50:27.129931 29562 raft_consensus.cc:2681] T 898caf6f94024100b52a3e03009354ff P 7ba117825c9e4c9f9fa8eb9fb9a4efad: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 20:50:27.134392 30032 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.134572 30032 ts_tablet_manager.cc:1397] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.018s
I20250115 20:50:27.135325 30032 raft_consensus.cc:357] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.136219 30032 raft_consensus.cc:383] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.136298 30032 raft_consensus.cc:738] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.136564 30032 consensus_queue.cc:260] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.137319 30032 ts_tablet_manager.cc:1428] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250115 20:50:27.137643 30032 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.138787 29299 master_service.cc:946] ReplaceTablet: received request to replace tablet db481c53f6d3402fb65a20c172db6f85 from {username='slave'} at 127.0.0.1:45056
I20250115 20:50:27.139319 30032 tablet_bootstrap.cc:654] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.144258 29296 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_cf9040b1c4a34c71aee275e501cd7a90" } modify_external_catalogs: true
I20250115 20:50:27.144495 29296 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_cf9040b1c4a34c71aee275e501cd7a90" } modify_external_catalogs: true
I20250115 20:50:27.146664 30034 tablet_bootstrap.cc:492] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.146816 30034 ts_tablet_manager.cc:1397] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20250115 20:50:27.147555 30034 raft_consensus.cc:357] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.147747 30034 raft_consensus.cc:383] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.147828 30034 raft_consensus.cc:738] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.148078 30034 consensus_queue.cc:260] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.150413 30034 ts_tablet_manager.cc:1428] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 20:50:27.151707 30034 tablet_bootstrap.cc:492] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:27.153681 30034 tablet_bootstrap.cc:654] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.153923 30032 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.154062 30032 ts_tablet_manager.cc:1397] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20250115 20:50:27.156369 30032 raft_consensus.cc:357] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.156652 30032 raft_consensus.cc:383] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.156772 30032 raft_consensus.cc:738] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.157090 30032 consensus_queue.cc:260] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.157864 30032 ts_tablet_manager.cc:1428] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250115 20:50:27.158128 30032 tablet_bootstrap.cc:492] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.159905 30032 tablet_bootstrap.cc:654] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
W20250115 20:50:27.161701 29301 catalog_manager.cc:5359] Ignoring report from unknown tablet 07b80aa3def740d6b6ba07698a9a6153
I20250115 20:50:27.161901 29299 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e7d519a9085e41eba7ed21d614a9ee86" } modify_external_catalogs: true
I20250115 20:50:27.162118 29299 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e7d519a9085e41eba7ed21d614a9ee86" } modify_external_catalogs: true
I20250115 20:50:27.162366 29301 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e3b5a2b22a3f4a768c449c7545648f1a" } modify_external_catalogs: true
I20250115 20:50:27.162633 29301 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e3b5a2b22a3f4a768c449c7545648f1a" } modify_external_catalogs: true
W20250115 20:50:27.167089 29300 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9f59fb888bd4c90b4f6f89671c71f1c consensus state has no leader
I20250115 20:50:27.167482 29300 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_92dfea87af06467d9ff104073cf60513" } new_table_name: "default.Table_70c940ffbcb84278857ca92ffc93b219" modify_external_catalogs: true
W20250115 20:50:27.168828 29295 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f481c13ed83e4eb0b42523c6c87dc49f consensus state has no leader
I20250115 20:50:27.169178 29295 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e2e1c0dd8e2646cfa6fab8f9fbc1bc15" } modify_external_catalogs: true
I20250115 20:50:27.169309 29295 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_e2e1c0dd8e2646cfa6fab8f9fbc1bc15" } modify_external_catalogs: true
I20250115 20:50:27.169368 30034 tablet_bootstrap.cc:492] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.169781 30032 tablet_bootstrap.cc:492] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.169888 30034 ts_tablet_manager.cc:1397] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250115 20:50:27.169936 30032 ts_tablet_manager.cc:1397] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250115 20:50:27.170469 29298 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet f481c13ed83e4eb0b42523c6c87dc49f
I20250115 20:50:27.170650 30032 raft_consensus.cc:357] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.170866 30032 raft_consensus.cc:383] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.170939 30032 raft_consensus.cc:738] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.170985 29298 catalog_manager.cc:6596] ReplaceTablet: tablet f481c13ed83e4eb0b42523c6c87dc49f deleted and replaced by tablet b5b9903ee35c441ba7f2c96463f70b84
I20250115 20:50:27.171144 30032 consensus_queue.cc:260] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.171362 29298 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:45056:
table { table_name: "default.Table_3815a22e9f88477a821e43581926909b" } new_table_name: "default.Table_06b3eeb678e8427dbada426371de2ffd" modify_external_catalogs: true
I20250115 20:50:27.171533 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f with delete_type TABLET_DATA_DELETED (replaced by tablet b5b9903ee35c441ba7f2c96463f70b84) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:27.171996 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f with delete_type TABLET_DATA_DELETED (replaced by tablet b5b9903ee35c441ba7f2c96463f70b84) from {username='slave'} at 127.0.0.1:37270
W20250115 20:50:27.172609 29297 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a57d470eb9be46d9a14f582b67e2e829 consensus state has no leader
W20250115 20:50:27.172755 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet f481c13ed83e4eb0b42523c6c87dc49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f481c13ed83e4eb0b42523c6c87dc49f already in progress: creating tablet
W20250115 20:50:27.173000 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet f481c13ed83e4eb0b42523c6c87dc49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f481c13ed83e4eb0b42523c6c87dc49f already in progress: creating tablet
I20250115 20:50:27.173132 29297 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_f116a1c3574b4e538458d2870e8abb48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:27.173527 30032 ts_tablet_manager.cc:1428] T 570c4a9054164d2982ee9cfe6c6589da P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250115 20:50:27.173719 29297 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f116a1c3574b4e538458d2870e8abb48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 20:50:27.173923 30032 tablet_bootstrap.cc:492] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.175223 29302 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet 80b8d0224d7443859c4dbc7dc8774863
I20250115 20:50:27.172564 30034 raft_consensus.cc:357] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.175446 30034 raft_consensus.cc:383] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.175493 30032 tablet_bootstrap.cc:654] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.175546 30034 raft_consensus.cc:738] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.176013 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:27.176002 29302 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "default.Table_06579dd6d834416ea135704694a4e553"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 20:50:27.176178 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:51536
W20250115 20:50:27.176481 29302 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06579dd6d834416ea135704694a4e553 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 20:50:27.176636 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet 80b8d0224d7443859c4dbc7dc8774863 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b8d0224d7443859c4dbc7dc8774863 already in progress: creating tablet
W20250115 20:50:27.176867 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet 80b8d0224d7443859c4dbc7dc8774863 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b8d0224d7443859c4dbc7dc8774863 already in progress: creating tablet
I20250115 20:50:27.175835 30034 consensus_queue.cc:260] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.179059 30034 ts_tablet_manager.cc:1428] T 24de8c78c961473297a1f3713f438031 P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
W20250115 20:50:27.180783 29294 sys_catalog.cc:609] row 1: Not found: key not found
W20250115 20:50:27.180986 29294 catalog_manager.cc:6561] Corruption: an error occurred while writing to the sys-catalog: failed to write one or more rows
I20250115 20:50:27.181030 30034 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
F20250115 20:50:27.181692 30007 master-stress-test.cc:391] Check failed: _s.ok() Bad status: Corruption: an error occurred while writing to the sys-catalog: failed to write one or more rows
I20250115 20:50:27.182511 30034 tablet_bootstrap.cc:654] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
*** Check failure stack trace: ***
*** Aborted at 1736974227 (unix time) try "date -d @1736974227" if you are using GNU date ***
I20250115 20:50:27.184257 30034 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.184387 30034 ts_tablet_manager.cc:1397] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250115 20:50:27.185122 30034 raft_consensus.cc:357] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.185312 30034 raft_consensus.cc:383] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.185397 30034 raft_consensus.cc:738] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.185520 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet f481c13ed83e4eb0b42523c6c87dc49f with delete_type TABLET_DATA_DELETED (replaced by tablet b5b9903ee35c441ba7f2c96463f70b84) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:27.185648 30034 consensus_queue.cc:260] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.186226 29296 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 3dfb97c4af714cb0add6d3dda0977312
I20250115 20:50:27.187726 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b8d0224d7443859c4dbc7dc8774863 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:49312
W20250115 20:50:27.188555 29822 tablet.cc:2367] T 07b80aa3def740d6b6ba07698a9a6153 P efbc16385f7f43bdad120af5153d3b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000724f) received by PID 29263 (TID 0x7f59e1a74700) from PID 29263; stack trace: ***
I20250115 20:50:27.190127 30034 ts_tablet_manager.cc:1428] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
@ 0x7f59fec76980 (unknown) at ??:0
@ 0x7f59f7c8dfb7 gsignal at ??:0
I20250115 20:50:27.193445 30034 tablet_bootstrap.cc:492] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:27.193908 30045 raft_consensus.cc:491] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 20:50:27.194260 29681 tablet.cc:2367] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 20:50:27.194129 30045 raft_consensus.cc:513] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
@ 0x7f59f7c8f921 abort at ??:0
@ 0x7f59fa9b6dcd google::LogMessage::Fail() at ??:0
I20250115 20:50:27.196178 30045 leader_election.cc:290] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569), 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
W20250115 20:50:27.197325 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet f481c13ed83e4eb0b42523c6c87dc49f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f481c13ed83e4eb0b42523c6c87dc49f already in progress: creating tablet
W20250115 20:50:27.197638 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet 80b8d0224d7443859c4dbc7dc8774863 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80b8d0224d7443859c4dbc7dc8774863 already in progress: creating tablet
@ 0x7f59fa9bab93 google::LogMessage::SendToLog() at ??:0
@ 0x7f59fa9b67cc google::LogMessage::Flush() at ??:0
@ 0x7f59fa9b7f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x40a885 kudu::MasterStressTest::ReplaceTabletThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:391
@ 0x447a84 std::__invoke_impl<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:73
@ 0x44793d std::__invoke<>() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95
W20250115 20:50:27.206924 29296 catalog_manager.cc:5359] Ignoring report from unknown tablet 24de8c78c961473297a1f3713f438031
W20250115 20:50:27.207048 29296 catalog_manager.cc:5359] Ignoring report from unknown tablet f175ae17b4e04f7a99b5ce55cdec12bf
W20250115 20:50:27.207731 29300 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c9bd162ce374885b0dcc8764d4752c3 consensus state has no leader
@ 0x4478fa _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTclsr3stdE8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:234
I20250115 20:50:27.214537 29300 master_service.cc:946] ReplaceTablet: received request to replace tablet a66dfb75e5df40a4b44950456464ffaf from {username='slave'} at 127.0.0.1:45056
@ 0x4478ae std::thread::_Invoker<>::operator()() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:243
@ 0x4475e9 std::thread::_State_impl<>::_M_run() at /usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/thread:186
I20250115 20:50:27.216174 29301 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet cfd32f2f4cb245379491ca1ed81f600c
@ 0x7f59f83156df (unknown) at ??:0
I20250115 20:50:27.217336 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd32f2f4cb245379491ca1ed81f600c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:27.217599 30037 tablet_replica.cc:331] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: stopping tablet replica
I20250115 20:50:27.217782 30037 raft_consensus.cc:2238] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:27.217933 30037 raft_consensus.cc:2267] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:27.218071 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd32f2f4cb245379491ca1ed81f600c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:27.218791 30037 ts_tablet_manager.cc:1905] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f59fec6b6db start_thread at ??:0
W20250115 20:50:27.219002 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet cfd32f2f4cb245379491ca1ed81f600c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cfd32f2f4cb245379491ca1ed81f600c already in progress: creating tablet
@ 0x7f59f7d7071f clone at ??:0
I20250115 20:50:27.222409 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd32f2f4cb245379491ca1ed81f600c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:27.222740 30038 tablet_replica.cc:331] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: stopping tablet replica
I20250115 20:50:27.222942 30038 raft_consensus.cc:2238] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:27.223102 30038 raft_consensus.cc:2267] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:27.223703 30038 ts_tablet_manager.cc:1905] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:27.225937 30037 ts_tablet_manager.cc:1918] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.226060 30037 log.cc:1198] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/wals/cfd32f2f4cb245379491ca1ed81f600c
I20250115 20:50:27.226111 29295 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 506da16b1be6409fa18289e9d6a3f23e
I20250115 20:50:27.226425 30037 ts_tablet_manager.cc:1939] T cfd32f2f4cb245379491ca1ed81f600c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting consensus metadata
I20250115 20:50:27.227447 29286 catalog_manager.cc:4872] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): tablet cfd32f2f4cb245379491ca1ed81f600c (table default.Table_e3b5a2b22a3f4a768c449c7545648f1a [id=6053566edd8b46e8879c858544edc22b]) successfully deleted
I20250115 20:50:27.222512 30034 tablet_bootstrap.cc:654] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.228470 30032 tablet_bootstrap.cc:492] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.228664 30032 ts_tablet_manager.cc:1397] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.006s
W20250115 20:50:27.229236 29298 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b074fd0be9343f99e97f762ad37d906 has no consensus state
I20250115 20:50:27.229354 30032 raft_consensus.cc:357] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.229699 30032 raft_consensus.cc:383] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.229784 30032 raft_consensus.cc:738] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.229811 30038 ts_tablet_manager.cc:1918] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.229933 30038 log.cc:1198] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-1/wal/wals/cfd32f2f4cb245379491ca1ed81f600c
I20250115 20:50:27.230015 30032 consensus_queue.cc:260] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.230260 30038 ts_tablet_manager.cc:1939] T cfd32f2f4cb245379491ca1ed81f600c P efbc16385f7f43bdad120af5153d3b90: Deleting consensus metadata
I20250115 20:50:27.230640 30032 ts_tablet_manager.cc:1428] T f9f59fb888bd4c90b4f6f89671c71f1c P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 20:50:27.231204 30032 tablet_bootstrap.cc:492] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.231638 30033 tablet_bootstrap.cc:492] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: No bootstrap required, opened a new log
I20250115 20:50:27.231782 30033 ts_tablet_manager.cc:1397] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: Time spent bootstrapping tablet: real 0.222s user 0.031s sys 0.056s
I20250115 20:50:27.234021 30032 tablet_bootstrap.cc:654] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.234578 29288 catalog_manager.cc:4872] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): tablet cfd32f2f4cb245379491ca1ed81f600c (table default.Table_e3b5a2b22a3f4a768c449c7545648f1a [id=6053566edd8b46e8879c858544edc22b]) successfully deleted
I20250115 20:50:27.234788 30033 raft_consensus.cc:357] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.235036 30033 raft_consensus.cc:383] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.235131 30033 raft_consensus.cc:738] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263078466c184592adf0cd2a8d6593d2, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.235357 30033 consensus_queue.cc:260] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.238550 29297 catalog_manager.cc:5618] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249) does not have the latest schema for tablet 16d2fc924b454a438e949b65a1b1575f (table default.Table_61db47e7fb0e4aa5a2528f57b57a74e0 [id=944bf40a4d9241fcb1323e0f749ee4cb]). Expected version 2 got 0
I20250115 20:50:27.243260 30034 tablet_bootstrap.cc:492] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.243391 30032 tablet_bootstrap.cc:492] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.243436 30034 ts_tablet_manager.cc:1397] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20250115 20:50:27.243575 30032 ts_tablet_manager.cc:1397] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20250115 20:50:27.244028 30033 ts_tablet_manager.cc:1428] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250115 20:50:27.244436 30032 raft_consensus.cc:357] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.244666 30032 raft_consensus.cc:383] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.244737 30032 raft_consensus.cc:738] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.244957 30032 consensus_queue.cc:260] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.245476 29297 catalog_manager.cc:5618] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249) does not have the latest schema for tablet 16d2fc924b454a438e949b65a1b1575f (table default.Table_61db47e7fb0e4aa5a2528f57b57a74e0 [id=944bf40a4d9241fcb1323e0f749ee4cb]). Expected version 2 got 0
I20250115 20:50:27.246182 30032 ts_tablet_manager.cc:1428] T b9a3282b69534422a7c462f32fae13ce P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 20:50:27.246639 30032 tablet_bootstrap.cc:492] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.246951 29299 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c96ad6fe132f49f5a0bb8b936073674c
I20250115 20:50:27.249061 30032 tablet_bootstrap.cc:654] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.250982 30033 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Bootstrap starting.
I20250115 20:50:27.253542 29300 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 43a8f945f14d4739b9fb165b5a945ab7: Sending DeleteTablet for 3 replicas of tablet a66dfb75e5df40a4b44950456464ffaf
I20250115 20:50:27.253975 29300 catalog_manager.cc:6596] ReplaceTablet: tablet a66dfb75e5df40a4b44950456464ffaf deleted and replaced by tablet a0d7b98a6f0c448e8877c40e93b4c202
I20250115 20:50:27.254374 30033 tablet_bootstrap.cc:654] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.254745 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet a66dfb75e5df40a4b44950456464ffaf with delete_type TABLET_DATA_DELETED (replaced by tablet a0d7b98a6f0c448e8877c40e93b4c202) from {username='slave'} at 127.0.0.1:37270
W20250115 20:50:27.255417 29286 catalog_manager.cc:4852] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): delete failed for tablet a66dfb75e5df40a4b44950456464ffaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a66dfb75e5df40a4b44950456464ffaf already in progress: creating tablet
I20250115 20:50:27.256758 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet a66dfb75e5df40a4b44950456464ffaf with delete_type TABLET_DATA_DELETED (replaced by tablet a0d7b98a6f0c448e8877c40e93b4c202) from {username='slave'} at 127.0.0.1:51536
W20250115 20:50:27.257463 29286 catalog_manager.cc:4852] TS 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): delete failed for tablet a66dfb75e5df40a4b44950456464ffaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a66dfb75e5df40a4b44950456464ffaf already in progress: creating tablet
I20250115 20:50:27.257551 29746 tablet_service.cc:1514] Processing DeleteTablet for tablet a66dfb75e5df40a4b44950456464ffaf with delete_type TABLET_DATA_DELETED (replaced by tablet a0d7b98a6f0c448e8877c40e93b4c202) from {username='slave'} at 127.0.0.1:49312
I20250115 20:50:27.259936 30041 raft_consensus.cc:491] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 20:50:27.260126 30041 raft_consensus.cc:513] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.260816 30041 leader_election.cc:290] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775), 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:27.260922 30034 raft_consensus.cc:357] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.261226 30034 raft_consensus.cc:383] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.261297 30034 raft_consensus.cc:738] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.261469 29912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d2fc924b454a438e949b65a1b1575f" candidate_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263078466c184592adf0cd2a8d6593d2" is_pre_election: true
I20250115 20:50:27.261562 30034 consensus_queue.cc:260] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.261840 29912 raft_consensus.cc:2463] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba117825c9e4c9f9fa8eb9fb9a4efad in term 0.
I20250115 20:50:27.262318 29560 leader_election.cc:304] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 263078466c184592adf0cd2a8d6593d2, 7ba117825c9e4c9f9fa8eb9fb9a4efad; no voters:
W20250115 20:50:27.262817 29288 catalog_manager.cc:4852] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775): delete failed for tablet a66dfb75e5df40a4b44950456464ffaf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a66dfb75e5df40a4b44950456464ffaf already in progress: creating tablet
I20250115 20:50:27.263053 29771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d2fc924b454a438e949b65a1b1575f" candidate_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbc16385f7f43bdad120af5153d3b90" is_pre_election: true
I20250115 20:50:27.263437 29771 raft_consensus.cc:2463] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ba117825c9e4c9f9fa8eb9fb9a4efad in term 0.
I20250115 20:50:27.264670 30041 raft_consensus.cc:2798] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 20:50:27.264801 30041 raft_consensus.cc:491] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 20:50:27.264897 30041 raft_consensus.cc:3054] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Advancing to term 1
I20250115 20:50:27.267849 30041 raft_consensus.cc:513] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
W20250115 20:50:27.268210 29300 catalog_manager.cc:5359] Ignoring report from unknown tablet 570c4a9054164d2982ee9cfe6c6589da
I20250115 20:50:27.269205 29771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d2fc924b454a438e949b65a1b1575f" candidate_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbc16385f7f43bdad120af5153d3b90"
I20250115 20:50:27.269383 29771 raft_consensus.cc:3054] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Advancing to term 1
I20250115 20:50:27.271188 30034 ts_tablet_manager.cc:1428] T 570c4a9054164d2982ee9cfe6c6589da P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20250115 20:50:27.272058 29771 raft_consensus.cc:2463] T 16d2fc924b454a438e949b65a1b1575f P efbc16385f7f43bdad120af5153d3b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba117825c9e4c9f9fa8eb9fb9a4efad in term 1.
I20250115 20:50:27.272836 29912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d2fc924b454a438e949b65a1b1575f" candidate_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "263078466c184592adf0cd2a8d6593d2"
I20250115 20:50:27.273052 29912 raft_consensus.cc:3054] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Advancing to term 1
I20250115 20:50:27.275099 29562 leader_election.cc:304] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ba117825c9e4c9f9fa8eb9fb9a4efad, efbc16385f7f43bdad120af5153d3b90; no voters:
I20250115 20:50:27.276096 30069 raft_consensus.cc:2798] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 1 FOLLOWER]: Leader election won for term 1
I20250115 20:50:27.276751 29912 raft_consensus.cc:2463] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ba117825c9e4c9f9fa8eb9fb9a4efad in term 1.
I20250115 20:50:27.277658 30032 tablet_bootstrap.cc:492] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
W20250115 20:50:27.277781 29300 catalog_manager.cc:5359] Ignoring report from unknown tablet 570c4a9054164d2982ee9cfe6c6589da
I20250115 20:50:27.277812 30032 ts_tablet_manager.cc:1397] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20250115 20:50:27.278627 30032 raft_consensus.cc:357] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.278811 30032 raft_consensus.cc:383] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.278903 30032 raft_consensus.cc:738] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.279384 30041 leader_election.cc:290] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [CANDIDATE]: Term 1 election: Requested vote from peers efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775), 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249)
I20250115 20:50:27.279650 30034 tablet_bootstrap.cc:492] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:27.279114 30032 consensus_queue.cc:260] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.280476 30032 ts_tablet_manager.cc:1428] T 96b3c24367b84205a25f6ea54c084564 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 20:50:27.280725 30032 tablet_bootstrap.cc:492] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.282070 30034 tablet_bootstrap.cc:654] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.282454 30032 tablet_bootstrap.cc:654] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.283166 30069 raft_consensus.cc:695] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 1 LEADER]: Becoming Leader. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Running, Role: LEADER
I20250115 20:50:27.285153 30069 consensus_queue.cc:237] T 16d2fc924b454a438e949b65a1b1575f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
W20250115 20:50:27.304333 29704 leader_election.cc:343] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7ba117825c9e4c9f9fa8eb9fb9a4efad (127.28.147.193:41569): Not found: Tablet not found: f175ae17b4e04f7a99b5ce55cdec12bf
I20250115 20:50:27.304484 30032 tablet_bootstrap.cc:492] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.304709 30032 ts_tablet_manager.cc:1397] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.021s
I20250115 20:50:27.305531 30032 raft_consensus.cc:357] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.305784 30032 raft_consensus.cc:383] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.305860 30032 raft_consensus.cc:738] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.306524 30032 consensus_queue.cc:260] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.307202 30032 ts_tablet_manager.cc:1428] T f481c13ed83e4eb0b42523c6c87dc49f P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 20:50:27.307512 30032 tablet_bootstrap.cc:492] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.309979 29613 tablet_service.cc:1514] Processing DeleteTablet for tablet 07b80aa3def740d6b6ba07698a9a6153 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b3c24367b84205a25f6ea54c084564) from {username='slave'} at 127.0.0.1:51536
I20250115 20:50:27.310117 30032 tablet_bootstrap.cc:654] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.310302 30037 tablet_replica.cc:331] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: stopping tablet replica
I20250115 20:50:27.310513 30037 raft_consensus.cc:2238] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:27.310715 30037 raft_consensus.cc:2267] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:27.311496 30037 ts_tablet_manager.cc:1905] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:27.317421 30034 tablet_bootstrap.cc:492] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90: No bootstrap required, opened a new log
I20250115 20:50:27.317521 30032 tablet_bootstrap.cc:492] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.317588 30034 ts_tablet_manager.cc:1397] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90: Time spent bootstrapping tablet: real 0.038s user 0.000s sys 0.004s
I20250115 20:50:27.317663 30032 ts_tablet_manager.cc:1397] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20250115 20:50:27.318177 30037 ts_tablet_manager.cc:1918] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.318284 30037 log.cc:1198] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-0/wal/wals/07b80aa3def740d6b6ba07698a9a6153
I20250115 20:50:27.318363 30034 raft_consensus.cc:357] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.318639 30034 raft_consensus.cc:383] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.318760 30034 raft_consensus.cc:738] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbc16385f7f43bdad120af5153d3b90, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.319059 30034 consensus_queue.cc:260] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.320281 30033 tablet_bootstrap.cc:492] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: No bootstrap required, opened a new log
I20250115 20:50:27.320457 30033 ts_tablet_manager.cc:1397] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.022s
I20250115 20:50:27.321493 30033 raft_consensus.cc:357] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.321724 30033 raft_consensus.cc:383] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.321799 30033 raft_consensus.cc:738] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 263078466c184592adf0cd2a8d6593d2, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.322057 30033 consensus_queue.cc:260] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.322595 30033 ts_tablet_manager.cc:1428] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 20:50:27.325681 29889 tablet_service.cc:1514] Processing DeleteTablet for tablet cfd32f2f4cb245379491ca1ed81f600c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 20:50:27 UTC) from {username='slave'} at 127.0.0.1:37270
I20250115 20:50:27.325953 30039 tablet_replica.cc:331] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: stopping tablet replica
I20250115 20:50:27.326164 30039 raft_consensus.cc:2238] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 20:50:27.326347 30039 raft_consensus.cc:2267] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 20:50:27.326673 30034 ts_tablet_manager.cc:1428] T f9f59fb888bd4c90b4f6f89671c71f1c P efbc16385f7f43bdad120af5153d3b90: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 20:50:27.327687 30034 tablet_bootstrap.cc:492] T b9a3282b69534422a7c462f32fae13ce P efbc16385f7f43bdad120af5153d3b90: Bootstrap starting.
I20250115 20:50:27.328310 30037 ts_tablet_manager.cc:1939] T 07b80aa3def740d6b6ba07698a9a6153 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Deleting consensus metadata
I20250115 20:50:27.329406 30032 raft_consensus.cc:357] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.329631 30034 tablet_bootstrap.cc:654] T b9a3282b69534422a7c462f32fae13ce P efbc16385f7f43bdad120af5153d3b90: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.329644 30032 raft_consensus.cc:383] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.329743 30032 raft_consensus.cc:738] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.330001 30032 consensus_queue.cc:260] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } } peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } }
I20250115 20:50:27.333374 30033 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P 263078466c184592adf0cd2a8d6593d2: Bootstrap starting.
I20250115 20:50:27.333681 30032 ts_tablet_manager.cc:1428] T 8db28a3ed90248b69b7e4934d9e124d3 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250115 20:50:27.334025 30032 tablet_bootstrap.cc:492] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.335160 30033 tablet_bootstrap.cc:654] T 07b80aa3def740d6b6ba07698a9a6153 P 263078466c184592adf0cd2a8d6593d2: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.336571 30032 tablet_bootstrap.cc:654] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.337607 30039 ts_tablet_manager.cc:1905] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 20:50:27.344235 30039 ts_tablet_manager.cc:1918] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 20:50:27.344368 30039 log.cc:1198] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Deleting WAL directory at /tmp/dist-test-taskLfmlVi/test-tmp/master-stress-test.1.CatalogManagerConfigurations_MasterStressTest.Test_1.1736974222973881-29263-0/minicluster-data/ts-2/wal/wals/cfd32f2f4cb245379491ca1ed81f600c
I20250115 20:50:27.345136 30032 tablet_bootstrap.cc:492] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: No bootstrap required, opened a new log
I20250115 20:50:27.345295 30032 ts_tablet_manager.cc:1397] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20250115 20:50:27.345464 30039 ts_tablet_manager.cc:1939] T cfd32f2f4cb245379491ca1ed81f600c P 263078466c184592adf0cd2a8d6593d2: Deleting consensus metadata
I20250115 20:50:27.346166 30032 raft_consensus.cc:357] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } }
I20250115 20:50:27.346402 30032 raft_consensus.cc:383] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 20:50:27.346482 30032 raft_consensus.cc:738] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba117825c9e4c9f9fa8eb9fb9a4efad, State: Initialized, Role: FOLLOWER
I20250115 20:50:27.346503 29286 catalog_manager.cc:4872] TS 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): tablet cfd32f2f4cb245379491ca1ed81f600c (table default.Table_e3b5a2b22a3f4a768c449c7545648f1a [id=6053566edd8b46e8879c858544edc22b]) successfully deleted
I20250115 20:50:27.346817 30032 consensus_queue.cc:260] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbc16385f7f43bdad120af5153d3b90" member_type: VOTER last_known_addr { host: "127.28.147.194" port: 46775 } } peers { permanent_uuid: "263078466c184592adf0cd2a8d6593d2" member_type: VOTER last_known_addr { host: "127.28.147.195" port: 39249 } } peers { permanent_uuid: "7ba117825c9e4c9f9fa8eb9fb9a4efad" member_type: VOTER last_known_addr { host: "127.28.147.193" port: 41569 } }
I20250115 20:50:27.347349 30032 ts_tablet_manager.cc:1428] T 80b8d0224d7443859c4dbc7dc8774863 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 20:50:27.347699 30032 tablet_bootstrap.cc:492] T 0c76efc8ea844a4e975bc064e7efb402 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Bootstrap starting.
I20250115 20:50:27.349365 30032 tablet_bootstrap.cc:654] T 0c76efc8ea844a4e975bc064e7efb402 P 7ba117825c9e4c9f9fa8eb9fb9a4efad: Neither blocks nor log segments found. Creating new log.
I20250115 20:50:27.351213 29985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 43a8f945f14d4739b9fb165b5a945ab7)
I20250115 20:50:27.351375 29985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43a8f945f14d4739b9fb165b5a945ab7" member_type: VOTER last_known_addr { host: "127.28.147.254" port: 45409 } } peers { permanent_uuid: "af9e84ab837a4c4face479825e0bded7" member_type: VOTER last_known_addr { host: "127.28.147.253" port: 46053 } } peers { permanent_uuid: "1195eca7127442e7a2c5a9e4df7b679a" member_type: VOTER last_known_addr { host: "127.28.147.252" port: 38139 } }
I20250115 20:50:27.353248 29311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "43a8f945f14d4739b9fb165b5a945ab7" is_pre_election: true
I20250115 20:50:27.353811 29385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1195eca7127442e7a2c5a9e4df7b679a" candidate_term: 5 candidate_status { last_received { term: 4 index: 28 } } ignore_live_leader: false dest_uuid: "af9e84ab837a4c4face479825e0bded7" is_pre_election: true
I20250115 20:50:27.354828 30033 tablet_bootstrap.cc:492] T 07b80aa3def740d6b6ba07698a9a6153 P 263078466c184592adf0cd2a8d6593d2: No bootstrap required, opened a new log
I20250115 20:50:27.354974 30033 ts_tablet_manager.cc:1397] T 07b80aa3def740d6b6ba07698a9a6153 P 263078466c184592adf0cd2a8d6593d2: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20250115 20:50:27.352476 29985 leader_election.cc:290] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 43a8f945f14d4739b9fb165b5a945ab7 (127.28.147.254:45409), af9e84ab837a4c4face479825e0bded7 (127.28.147.253:46053)
I20250115 20:50:27.355623 29299 catalog_manager.cc:5618] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775) does not have the latest schema for tablet f9f59fb888bd4c90b4f6f89671c71f1c (table default.Table_1f38bf9b3a2540b89cfcbf6a05791b79 [id=67f6e4ad0f04425bb8a1f70fe3d50275]). Expected version 1 got 0
I20250115 20:50:27.358098 29613 tablet_service.cc:1467] Processing CreateTablet for tablet 486e22efb3b144f2afbce8087ea11f09 (DEFAULT_TABLE table=default.Table_06579dd6d834416ea135704694a4e553 [id=7d49ef758f8f4067b2c2d96b9e4395aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.358520 29613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486e22efb3b144f2afbce8087ea11f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.361426 29746 tablet_service.cc:1467] Processing CreateTablet for tablet 486e22efb3b144f2afbce8087ea11f09 (DEFAULT_TABLE table=default.Table_06579dd6d834416ea135704694a4e553 [id=7d49ef758f8f4067b2c2d96b9e4395aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.361862 29746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486e22efb3b144f2afbce8087ea11f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.363678 29299 catalog_manager.cc:5618] TS efbc16385f7f43bdad120af5153d3b90 (127.28.147.194:46775) does not have the latest schema for tablet f9f59fb888bd4c90b4f6f89671c71f1c (table default.Table_1f38bf9b3a2540b89cfcbf6a05791b79 [id=67f6e4ad0f04425bb8a1f70fe3d50275]). Expected version 1 got 0
I20250115 20:50:27.365338 29610 tablet_service.cc:1467] Processing CreateTablet for tablet 41c84d7f61b340adb76ace371bc2263c (DEFAULT_TABLE table=default.Table_f116a1c3574b4e538458d2870e8abb48 [id=8156d55cbeef41dead00743d22b77f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.365731 29610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c84d7f61b340adb76ace371bc2263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.366938 29606 tablet_service.cc:1467] Processing CreateTablet for tablet f1a6950f0c8c437abf0796b22ec5f7f6 (DEFAULT_TABLE table=default.Table_21516e6dde08453997dbc7d7d7de4c2c [id=b47b83fa3f06447e8d4e3746de9c3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.367316 29606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6950f0c8c437abf0796b22ec5f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.369562 29889 tablet_service.cc:1467] Processing CreateTablet for tablet 486e22efb3b144f2afbce8087ea11f09 (DEFAULT_TABLE table=default.Table_06579dd6d834416ea135704694a4e553 [id=7d49ef758f8f4067b2c2d96b9e4395aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.370007 29885 tablet_service.cc:1467] Processing CreateTablet for tablet f1a6950f0c8c437abf0796b22ec5f7f6 (DEFAULT_TABLE table=default.Table_21516e6dde08453997dbc7d7d7de4c2c [id=b47b83fa3f06447e8d4e3746de9c3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.354382 29432 leader_election.cc:304] T 00000000000000000000000000000000 P 1195eca7127442e7a2c5a9e4df7b679a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1195eca7127442e7a2c5a9e4df7b679a; no voters: 43a8f945f14d4739b9fb165b5a945ab7, af9e84ab837a4c4face479825e0bded7
I20250115 20:50:27.372455 29887 tablet_service.cc:1467] Processing CreateTablet for tablet 41c84d7f61b340adb76ace371bc2263c (DEFAULT_TABLE table=default.Table_f116a1c3574b4e538458d2870e8abb48 [id=8156d55cbeef41dead00743d22b77f07]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 20:50:27.377717 29704 leader_election.cc:343] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 263078466c184592adf0cd2a8d6593d2 (127.28.147.195:39249): Not found: Tablet not found: f175ae17b4e04f7a99b5ce55cdec12bf
I20250115 20:50:27.377902 29745 tablet_service.cc:1467] Processing CreateTablet for tablet 41c84d7f61b340adb76ace371bc2263c (DEFAULT_TABLE table=default.Table_f116a1c3574b4e538458d2870e8abb48 [id=8156d55cbeef41dead00743d22b77f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.378824 29892 tablet_service.cc:1467] Processing CreateTablet for tablet 503776113a0c412e88c3f5dd6bfbf1de (DEFAULT_TABLE table=default.Table_028a64030c8141c89ccd3e63992c45ae [id=b31e6f16074f4b949fc7ca0b4d37d5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.382683 29750 tablet_service.cc:1467] Processing CreateTablet for tablet f1a6950f0c8c437abf0796b22ec5f7f6 (DEFAULT_TABLE table=default.Table_21516e6dde08453997dbc7d7d7de4c2c [id=b47b83fa3f06447e8d4e3746de9c3a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.383122 29750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6950f0c8c437abf0796b22ec5f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.384770 29704 leader_election.cc:304] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efbc16385f7f43bdad120af5153d3b90; no voters: 263078466c184592adf0cd2a8d6593d2, 7ba117825c9e4c9f9fa8eb9fb9a4efad
I20250115 20:50:27.385663 29889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486e22efb3b144f2afbce8087ea11f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.385942 30045 raft_consensus.cc:2743] T f175ae17b4e04f7a99b5ce55cdec12bf P efbc16385f7f43bdad120af5153d3b90 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 20:50:27.386178 29745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41c84d7f61b340adb76ace371bc2263c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.387738 29885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1a6950f0c8c437abf0796b22ec5f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.391062 29886 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9e550263f54282aff062dd52fb8ce0 (DEFAULT_TABLE table=default.Table_e80b327eb46e4273b48b6207e0806542 [id=2d759f8858fa47d290b155a97f646612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.391407 29607 tablet_service.cc:1467] Processing CreateTablet for tablet 6b074fd0be9343f99e97f762ad37d906 (DEFAULT_TABLE table=default.Table_06b3eeb678e8427dbada426371de2ffd [id=a5d5c66778954d96b6034982a2ffe9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.391836 29607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b074fd0be9343f99e97f762ad37d906. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.401389 29748 tablet_service.cc:1467] Processing CreateTablet for tablet 6b074fd0be9343f99e97f762ad37d906 (DEFAULT_TABLE table=default.Table_06b3eeb678e8427dbada426371de2ffd [id=a5d5c66778954d96b6034982a2ffe9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.401683 29751 tablet_service.cc:1467] Processing CreateTablet for tablet 503776113a0c412e88c3f5dd6bfbf1de (DEFAULT_TABLE table=default.Table_028a64030c8141c89ccd3e63992c45ae [id=b31e6f16074f4b949fc7ca0b4d37d5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.403280 29749 tablet_service.cc:1467] Processing CreateTablet for tablet cbd2cfb847474e2cb9840cc34cf8f83d (DEFAULT_TABLE table=default.Table_a749ae25fbf344fbba58ed4ea747addf [id=130839d83250467d9e4eda999c466156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.403704 29747 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9e550263f54282aff062dd52fb8ce0 (DEFAULT_TABLE table=default.Table_e80b327eb46e4273b48b6207e0806542 [id=2d759f8858fa47d290b155a97f646612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.404102 29891 tablet_service.cc:1467] Processing CreateTablet for tablet 6b074fd0be9343f99e97f762ad37d906 (DEFAULT_TABLE table=default.Table_06b3eeb678e8427dbada426371de2ffd [id=a5d5c66778954d96b6034982a2ffe9f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.404198 29744 tablet_service.cc:1467] Processing CreateTablet for tablet f95b5d5040ad445cbc4198f1acf622a4 (DEFAULT_TABLE table=default.Table_6509ddc7e973469799588a103285c235 [id=78d4702f31f543bcb63eee43ae675ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.404466 29748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b074fd0be9343f99e97f762ad37d906. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.404636 29890 tablet_service.cc:1467] Processing CreateTablet for tablet cbd2cfb847474e2cb9840cc34cf8f83d (DEFAULT_TABLE table=default.Table_a749ae25fbf344fbba58ed4ea747addf [id=130839d83250467d9e4eda999c466156]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 20:50:27.405088 29963 tablet.cc:2367] T 16d2fc924b454a438e949b65a1b1575f P 263078466c184592adf0cd2a8d6593d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 20:50:27.405095 29743 tablet_service.cc:1467] Processing CreateTablet for tablet 3563b13f9dff44999e3e118cf512b7f1 (DEFAULT_TABLE table=default.Table_ed3830fa6620490e8ef9969895bccd99 [id=7614a8e98b9f4e28bfe4f02e758b2855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.405813 29747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b9e550263f54282aff062dd52fb8ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.406144 29749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd2cfb847474e2cb9840cc34cf8f83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.406558 29612 tablet_service.cc:1467] Processing CreateTablet for tablet cbd2cfb847474e2cb9840cc34cf8f83d (DEFAULT_TABLE table=default.Table_a749ae25fbf344fbba58ed4ea747addf [id=130839d83250467d9e4eda999c466156]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.406942 29612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd2cfb847474e2cb9840cc34cf8f83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.408720 29751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 503776113a0c412e88c3f5dd6bfbf1de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.411482 29744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f95b5d5040ad445cbc4198f1acf622a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.412937 29743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3563b13f9dff44999e3e118cf512b7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.412951 29609 tablet_service.cc:1467] Processing CreateTablet for tablet f95b5d5040ad445cbc4198f1acf622a4 (DEFAULT_TABLE table=default.Table_6509ddc7e973469799588a103285c235 [id=78d4702f31f543bcb63eee43ae675ce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.413353 29609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f95b5d5040ad445cbc4198f1acf622a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.416813 29611 tablet_service.cc:1467] Processing CreateTablet for tablet 3563b13f9dff44999e3e118cf512b7f1 (DEFAULT_TABLE table=default.Table_ed3830fa6620490e8ef9969895bccd99 [id=7614a8e98b9f4e28bfe4f02e758b2855]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 20:50:27.417203 29611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3563b13f9dff44999e3e118cf512b7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 20:50:27.420879 29608 tablet_service.cc:1467] Processing CreateTablet for tablet b5b9903ee35c441ba7f2c96463f70b84 (DEFAULT_TABLE table=default.Table_c6ab6e8deb8549d18b9933ee083505ec [id=0f48512293574a0980c88eb48ea7b886]), partition=RANGE (key) PARTITION UNBOUNDED