Note: This is test shard 3 of 3.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:57:09.538046 14720 test_util.cc:268] Using random seed: -464698134
2024-05-01T19:57:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:57:09Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknpHgT3/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknpHgT3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/security.properties
2024-05-01 19:57:15: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknpHgT3/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasknpHgT3/thirdparty/src/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20240501 19:57:24.532485 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--webserver_interface=127.14.96.62
--webserver_port=0
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:24.647925 14962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:24.648195 14962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:24.648236 14962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:24.651439 14962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:24.651518 14962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:24.651538 14962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:24.651556 14962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:24.655395 14962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.96.62
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.14962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:24.656415 14962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:24.657594 14962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:24.665906 14968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:24.666453 14967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:24.667222 14970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:24.667883 14962 server_base.cc:1034] running on GCE node
I20240501 19:57:24.668437 14962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:24.669461 14962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:24.670615 14962 hybrid_clock.cc:648] HybridClock initialized: now 1714593444670605 us; error 28 us; skew 500 ppm
I20240501 19:57:24.678766 14962 webserver.cc:460] Webserver started at http://127.14.96.62:42813/ using document root <none> and password file <none>
I20240501 19:57:24.679402 14962 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:24.679486 14962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:24.679702 14962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:24.681535 14962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/instance:
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.682013 14962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal/instance:
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.685501 14962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:24.687752 14976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:24.688823 14962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:24.688973 14962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.689061 14962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:24.702908 14962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:24.703608 14962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:24.703773 14962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:24.710954 14962 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.62:37747
I20240501 19:57:24.710963 15028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.62:37747 every 8 connection(s)
I20240501 19:57:24.711979 14962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:24.714627 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.718611 15029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:24.719487 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 14962
I20240501 19:57:24.719638 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal/instance
I20240501 19:57:24.721318 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--webserver_interface=127.14.96.61
--webserver_port=0
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:24.730827 15029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:57:24.731627 14977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:24.732491 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:24.781733 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } attempt: 1
W20240501 19:57:24.783224 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:24.835606 15032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:24.835870 15032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:24.835913 15032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:24.839128 15032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:24.839197 15032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:24.839218 15032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:24.839236 15032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:24.843066 15032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.96.61
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:24.844071 15032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:24.845283 15032 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:24.851428 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } attempt: 2
W20240501 19:57:24.853073 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:24.853895 15038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:24.854516 15039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:24.855064 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:24.855739 15032 server_base.cc:1034] running on GCE node
I20240501 19:57:24.856297 15032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:24.857301 15032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:24.858460 15032 hybrid_clock.cc:648] HybridClock initialized: now 1714593444858445 us; error 31 us; skew 500 ppm
I20240501 19:57:24.860414 15032 webserver.cc:460] Webserver started at http://127.14.96.61:42603/ using document root <none> and password file <none>
I20240501 19:57:24.861027 15032 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:24.861089 15032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:24.861284 15032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:24.863008 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/instance:
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.863487 15032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal/instance:
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.866906 15032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:57:24.869282 15047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:24.870332 15032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:24.870445 15032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:24.870534 15032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:24.891963 15032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:24.892673 15032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:24.892874 15032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:24.899644 15099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.61:46321 every 8 connection(s)
I20240501 19:57:24.899647 15032 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.61:46321
I20240501 19:57:24.900730 15032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
I20240501 19:57:24.903388 15100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:24.907368 15100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:24.907729 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15032
I20240501 19:57:24.907836 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal/instance
I20240501 19:57:24.908640 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--webserver_interface=127.14.96.60
--webserver_port=0
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:24.917965 15100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:24.925020 15100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:57:24.925824 15049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:24.926659 15100 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:24.944300 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } attempt: 3
I20240501 19:57:24.949292 15029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:57:24.950618 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:24.975883 15100 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 1
W20240501 19:57:24.977488 15100 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:24.981781 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 1
W20240501 19:57:24.983110 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:25.028930 15101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:25.029214 15101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:25.029300 15101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:25.032568 15101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:25.032657 15101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:25.032694 15101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:25.032727 15101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:25.036933 15101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.96.60
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:25.037973 15101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:25.039170 15101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:25.045689 15100 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 2
W20240501 19:57:25.047008 15100 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:25.047106 15112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.047609 15110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.048249 15109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:25.048930 15101 server_base.cc:1034] running on GCE node
I20240501 19:57:25.049496 15101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:25.050467 15101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:25.051637 15101 hybrid_clock.cc:648] HybridClock initialized: now 1714593445051626 us; error 32 us; skew 500 ppm
I20240501 19:57:25.053741 15101 webserver.cc:460] Webserver started at http://127.14.96.60:43311/ using document root <none> and password file <none>
I20240501 19:57:25.054324 15101 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:25.054389 15101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:25.054597 15101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:25.056349 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/instance:
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.056869 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal/instance:
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.058264 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 2
W20240501 19:57:25.059552 15029 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.14.96.60:40075: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:25.060614 15101 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:57:25.063016 15118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.064088 15101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:25.064204 15101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.064291 15101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:25.077415 15101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:25.078063 15101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:25.078217 15101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:25.085484 15170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.60:40075 every 8 connection(s)
I20240501 19:57:25.085479 15101 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.60:40075
I20240501 19:57:25.086581 15101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
I20240501 19:57:25.089287 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:25.092916 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:25.096415 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15101
I20240501 19:57:25.096520 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal/instance
I20240501 19:57:25.102495 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:25.107514 15171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:57:25.116262 15171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap starting.
I20240501 19:57:25.118422 15171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:25.119598 15171 log.cc:824] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:25.121503 15171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: No bootstrap required, opened a new log
I20240501 19:57:25.124581 15171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.124893 15171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:25.125010 15171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Initialized, Role: FOLLOWER
I20240501 19:57:25.125487 15171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.125923 15179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.126036 15179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.126559 15171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:57:25.132143 15193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:57:25.132352 15193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:57:25.130733 15101 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: OK, instance_id { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" instance_seqno: 1714593444709122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 19:57:25.133232 15101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:25.138206 15100 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 3
I20240501 19:57:25.146368 15100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap starting.
I20240501 19:57:25.149199 15100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:25.150557 15100 log.cc:824] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:25.152516 15100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: No bootstrap required, opened a new log
I20240501 19:57:25.156064 15100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.156280 15100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:25.156349 15100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Initialized, Role: FOLLOWER
I20240501 19:57:25.156981 15100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.158365 15100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:25.158682 15029 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } attempt: 3
I20240501 19:57:25.163169 15032 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: OK, instance_id { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" instance_seqno: 1714593444709122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240501 19:57:25.165863 15032 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:25.167219 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap starting.
I20240501 19:57:25.170475 15029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:25.170900 15211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20240501 19:57:25.171710 15029 log.cc:824] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Log is configured to *not* fsync() on all Append() calls
W20240501 19:57:25.171748 15211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:57:25.171629 15197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.171875 15197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.197320 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: No bootstrap required, opened a new log
I20240501 19:57:25.201068 15029 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.201356 15029 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:25.201467 15029 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Initialized, Role: FOLLOWER
I20240501 19:57:25.202031 15029 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.202515 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.202719 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.203315 15029 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:57:25.205818 14982 tablet.cc:2369] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:57:25.212455 15229 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:57:25.212553 15229 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:57:25.215461 14962 master_runner.cc:263] No leader master found from master e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:25.215543 14962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 19:57:25.329484 15124 tablet.cc:2369] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:25.353519 15179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:25.353756 15179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.354954 15179 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:25.355957 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:25.356297 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 0.
I20240501 19:57:25.357586 15197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:25.357753 15197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.358047 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
I20240501 19:57:25.358247 15074 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 0.
I20240501 19:57:25.358759 15197 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:25.359369 15121 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:25.360370 15179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:25.360461 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:25.360428 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:25.360579 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d8e36923de4d1e8a6de93e73c84658 in term 0.
I20240501 19:57:25.360448 15179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:25.360785 15179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:25.360915 15050 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:25.361148 15197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:25.361217 15197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:25.361253 15197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:25.361971 15179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.362205 15145 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72d8e36923de4d1e8a6de93e73c84658 in current term 1: Already voted for candidate 62106a6dbb3846df9d3b29ab99fb3763 in this term.
I20240501 19:57:25.362193 15197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.362547 15197 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 1 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:25.363050 15145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763"
I20240501 19:57:25.363168 15145 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 72d8e36923de4d1e8a6de93e73c84658 in current term 1: Already voted for candidate 62106a6dbb3846df9d3b29ab99fb3763 in this term.
I20240501 19:57:25.363480 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:25.363590 15003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:25.363867 15179 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 1 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:25.364177 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658"
I20240501 19:57:25.364332 15074 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62106a6dbb3846df9d3b29ab99fb3763 in current term 1: Already voted for candidate 72d8e36923de4d1e8a6de93e73c84658 in this term.
I20240501 19:57:25.364671 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d8e36923de4d1e8a6de93e73c84658 in term 1.
I20240501 19:57:25.364979 15050 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763
I20240501 19:57:25.365168 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:25.365247 15197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:25.365340 15003 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 62106a6dbb3846df9d3b29ab99fb3763 in current term 1: Already voted for candidate 72d8e36923de4d1e8a6de93e73c84658 in this term.
I20240501 19:57:25.365443 15197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 LEADER]: Becoming Leader. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Running, Role: LEADER
I20240501 19:57:25.365710 15121 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:25.365754 15197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.365995 15179 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 19:57:25.367437 15197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.367621 15197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:25.375566 15234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:25.376636 15234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:25.392508 15145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Refusing update from remote peer 72d8e36923de4d1e8a6de93e73c84658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:25.392508 15003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Refusing update from remote peer 72d8e36923de4d1e8a6de93e73c84658: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:25.393122 15197 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:25.393577 15197 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:57:25.394382 15053 tablet.cc:2369] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:25.396833 15179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.396991 15179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.397233 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.397332 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.400182 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.400313 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:25.401567 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.401683 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.402201 15236 mvcc.cc:205] Tried to move back new op lower bound from 7022974752319967232 to 7022974752221081600. Current Snapshot: MvccSnapshot[applied={T|T < 7022974752319967232}]
I20240501 19:57:25.403852 15234 catalog_manager.cc:1347] Generated new cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:25.403923 15234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:25.406934 15197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.407056 15197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:25.408138 15179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:25.408247 15179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:25.408300 15237 mvcc.cc:205] Tried to move back new op lower bound from 7022974752319967232 to 7022974752221081600. Current Snapshot: MvccSnapshot[applied={T|T < 7022974752319967232}]
I20240501 19:57:25.438377 15234 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:25.440289 15234 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:25.459206 15234 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Generated new TSK 0
I20240501 19:57:25.460152 15234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:25.460820 15234 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:25.461293 15234 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: -1
I20240501 19:57:25.481251 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.96.1:0
--local_ip_for_outbound_sockets=127.14.96.1
--webserver_interface=127.14.96.1
--webserver_port=0
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20240501 19:57:25.594741 15244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:25.595002 15244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:25.595072 15244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:25.598340 15244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:25.598495 15244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.96.1
I20240501 19:57:25.602375 15244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.96.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.14.96.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--never_fsync=true
--heap_profile_path=/tmp/kudu.15244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.96.1
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:25.603358 15244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:25.604552 15244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:25.613879 15249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.613993 15250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.614744 15252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:25.615329 15244 server_base.cc:1034] running on GCE node
I20240501 19:57:25.615746 15244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:25.616436 15244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:25.617587 15244 hybrid_clock.cc:648] HybridClock initialized: now 1714593445617572 us; error 31 us; skew 500 ppm
I20240501 19:57:25.619474 15244 webserver.cc:460] Webserver started at http://127.14.96.1:45279/ using document root <none> and password file <none>
I20240501 19:57:25.620049 15244 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:25.620110 15244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:25.620306 15244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:25.622087 15244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data/instance:
uuid: "0c8dfc11dc7542f583644b9ac747545d"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.622578 15244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/instance:
uuid: "0c8dfc11dc7542f583644b9ac747545d"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.625965 15244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:25.628365 15258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.629555 15244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:25.629678 15244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal
uuid: "0c8dfc11dc7542f583644b9ac747545d"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.629786 15244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:25.660195 15244 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:25.660912 15244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:25.661101 15244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:25.661659 15244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:25.662554 15244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:25.662598 15244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.662639 15244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:25.662662 15244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.671850 15244 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.1:36247
I20240501 19:57:25.671873 15371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.1:36247 every 8 connection(s)
I20240501 19:57:25.672811 15244 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/data/info.pb
I20240501 19:57:25.677105 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15244
I20240501 19:57:25.677268 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/instance
I20240501 19:57:25.679844 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.96.2:0
--local_ip_for_outbound_sockets=127.14.96.2
--webserver_interface=127.14.96.2
--webserver_port=0
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240501 19:57:25.683295 15374 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:25.683310 15372 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:25.683568 15372 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.683960 15374 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.684161 15372 heartbeater.cc:511] Master 127.14.96.60:40075 requested a full tablet report, sending...
I20240501 19:57:25.684213 15374 heartbeater.cc:511] Master 127.14.96.61:46321 requested a full tablet report, sending...
I20240501 19:57:25.685701 15064 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:25.685699 15135 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:25.686184 15373 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:25.686257 15373 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.686409 15373 heartbeater.cc:511] Master 127.14.96.62:37747 requested a full tablet report, sending...
I20240501 19:57:25.687288 14992 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:25.687372 15064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.96.1:54707
W20240501 19:57:25.802166 15382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:25.802443 15382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:25.802528 15382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:25.805879 15382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:25.806053 15382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.96.2
I20240501 19:57:25.809872 15382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.96.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.14.96.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--never_fsync=true
--heap_profile_path=/tmp/kudu.15382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.96.2
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:25.810899 15382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:25.812072 15382 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:25.821060 15390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.821643 15387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:25.822221 15388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:25.822862 15382 server_base.cc:1034] running on GCE node
I20240501 19:57:25.823275 15382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:25.823928 15382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:25.825073 15382 hybrid_clock.cc:648] HybridClock initialized: now 1714593445825062 us; error 27 us; skew 500 ppm
I20240501 19:57:25.827050 15382 webserver.cc:460] Webserver started at http://127.14.96.2:34379/ using document root <none> and password file <none>
I20240501 19:57:25.827664 15382 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:25.827718 15382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:25.827913 15382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:25.829785 15382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data/instance:
uuid: "68539c0ecd3e44e080cb53987d4f4c23"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.830260 15382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/instance:
uuid: "68539c0ecd3e44e080cb53987d4f4c23"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.833782 15382 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:57:25.836069 15396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.837196 15382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:57:25.837301 15382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal
uuid: "68539c0ecd3e44e080cb53987d4f4c23"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:25.837396 15382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:25.859532 15382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:25.860198 15382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:25.860359 15382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:25.860965 15382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:25.861914 15382 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:25.861961 15382 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.862003 15382 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:25.862025 15382 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:25.871264 15382 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.2:34181
I20240501 19:57:25.871320 15509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.2:34181 every 8 connection(s)
I20240501 19:57:25.872254 15382 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/data/info.pb
I20240501 19:57:25.877544 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15382
I20240501 19:57:25.877671 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/instance
I20240501 19:57:25.882035 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.14.96.3:0
--local_ip_for_outbound_sockets=127.14.96.3
--webserver_interface=127.14.96.3
--webserver_port=0
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240501 19:57:25.882714 15512 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:25.882687 15511 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:25.882988 15512 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.882989 15511 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.883692 15512 heartbeater.cc:511] Master 127.14.96.61:46321 requested a full tablet report, sending...
I20240501 19:57:25.883692 15511 heartbeater.cc:511] Master 127.14.96.62:37747 requested a full tablet report, sending...
I20240501 19:57:25.884624 15510 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:25.884706 15510 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:25.884914 14992 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:25.884949 15510 heartbeater.cc:511] Master 127.14.96.60:40075 requested a full tablet report, sending...
I20240501 19:57:25.885315 15064 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:25.885922 15064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.96.2:56913
I20240501 19:57:25.886775 15135 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:25.959532 15518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:25.959729 15518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:25.960350 15518 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:25.960819 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
I20240501 19:57:25.960922 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:25.961333 15121 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:25.961632 15518 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 19:57:26.001431 15517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:26.001696 15517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:26.001749 15517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:26.005132 15517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:26.005326 15517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.14.96.3
I20240501 19:57:26.009361 15517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.14.96.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.14.96.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--never_fsync=true
--heap_profile_path=/tmp/kudu.15517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.14.96.3
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:26.010408 15517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:26.011631 15517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:26.020864 15524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:26.021450 15526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:26.021894 15523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:26.022567 15517 server_base.cc:1034] running on GCE node
I20240501 19:57:26.023037 15517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:26.023747 15517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:26.024920 15517 hybrid_clock.cc:648] HybridClock initialized: now 1714593446024903 us; error 32 us; skew 500 ppm
I20240501 19:57:26.026939 15517 webserver.cc:460] Webserver started at http://127.14.96.3:33323/ using document root <none> and password file <none>
I20240501 19:57:26.027559 15517 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:26.027626 15517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:26.027832 15517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:26.029702 15517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data/instance:
uuid: "7f011d422e164c949a9d54d846b7c432"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-spj6"
I20240501 19:57:26.030258 15517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/instance:
uuid: "7f011d422e164c949a9d54d846b7c432"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-spj6"
I20240501 19:57:26.033905 15517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:26.036509 15532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.037717 15517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:26.037856 15517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal
uuid: "7f011d422e164c949a9d54d846b7c432"
format_stamp: "Formatted at 2024-05-01 19:57:26 on dist-test-slave-spj6"
I20240501 19:57:26.037962 15517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:26.050673 15517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:26.051330 15517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:26.051510 15517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:26.052064 15517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:26.053175 15517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:26.053231 15517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.053294 15517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:26.053323 15517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:26.066066 15517 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.3:42023
I20240501 19:57:26.066082 15645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.3:42023 every 8 connection(s)
I20240501 19:57:26.067055 15517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/data/info.pb
I20240501 19:57:26.069237 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15517
I20240501 19:57:26.069362 14720 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/instance
I20240501 19:57:26.078646 15647 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:26.078835 15646 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:26.078922 15647 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:26.078920 15646 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:26.079665 15647 heartbeater.cc:511] Master 127.14.96.62:37747 requested a full tablet report, sending...
I20240501 19:57:26.079674 15646 heartbeater.cc:511] Master 127.14.96.60:40075 requested a full tablet report, sending...
I20240501 19:57:26.080058 15648 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:26.080142 15648 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:26.080351 15648 heartbeater.cc:511] Master 127.14.96.61:46321 requested a full tablet report, sending...
I20240501 19:57:26.080772 14992 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.081010 15064 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.081432 15135 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.081696 15064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.14.96.3:52705
I20240501 19:57:26.090759 14720 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240501 19:57:26.107061 14720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu with pid 15101
I20240501 19:57:26.109659 15061 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_31c7944e4b41499683c266b366d0711f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.109791 15063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_b78cbc9abde043b6a52288415772661c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.110256 15060 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_3c7dc746fd214065a974e0e8e2ec6f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.110569 15059 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_934e8fcbba9a4c2a9e875db16c2ff080"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.110826 15057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_40db4cea1aad411796b548b57c595842"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.111090 15056 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_a94d910a1db245728708bdc7eb0b50b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.111335 15058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_39fb93a866e54177a673e26dd8325964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.111603 15055 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_88be21f10fc74ddaab680d288fbb9a13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.109660 15062 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_0ca797a676d54ea89cfea3963a75b1d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:26.112691 15064 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_0c03b0e181d140c88754b558e4fb4393"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:57:26.152681 15060 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3c7dc746fd214065a974e0e8e2ec6f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152778 15058 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_39fb93a866e54177a673e26dd8325964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152895 15056 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a94d910a1db245728708bdc7eb0b50b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152979 15055 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_88be21f10fc74ddaab680d288fbb9a13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152681 15057 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_40db4cea1aad411796b548b57c595842 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152786 15063 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b78cbc9abde043b6a52288415772661c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.152691 15059 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_934e8fcbba9a4c2a9e875db16c2ff080 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:26.188102 15372 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:26.189900 15374 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
I20240501 19:57:26.213973 15229 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:26.214056 15229 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: loading cluster ID for follower catalog manager: success
I20240501 19:57:26.216084 15229 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: acquiring CA information for follower catalog manager: success
I20240501 19:57:26.217726 15229 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:26.388867 15512 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
W20240501 19:57:26.390046 15510 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:26.419072 15049 consensus_peers.cc:485] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 -> Peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Couldn't send request to peer 62106a6dbb3846df9d3b29ab99fb3763. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:26.447770 15696 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:26.447925 15696 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:26.448940 15696 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:26.449425 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
W20240501 19:57:26.450450 14978 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:26.450752 14977 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:26.450956 15696 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:26.575424 15060 catalog_manager.cc:2145] created HMS catalog entry for table [id=608d4b3e2b584ef8aac50ccdec82ca60]
I20240501 19:57:26.583330 15648 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
I20240501 19:57:26.588537 15060 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_3c7dc746fd214065a974e0e8e2ec6f28" } new_table_name: "default.Table_8060d3196acf479897cfe5f6f547dba9" modify_external_catalogs: true
W20240501 19:57:26.596673 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:26.600322 15444 tablet_service.cc:1467] Processing CreateTablet for tablet f12a4b397b8f4dedae851f98a8b18feb (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.601424 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12a4b397b8f4dedae851f98a8b18feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.609115 15580 tablet_service.cc:1467] Processing CreateTablet for tablet f12a4b397b8f4dedae851f98a8b18feb (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.610173 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12a4b397b8f4dedae851f98a8b18feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.612141 15055 catalog_manager.cc:2145] created HMS catalog entry for table [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:26.618324 15306 tablet_service.cc:1467] Processing CreateTablet for tablet f12a4b397b8f4dedae851f98a8b18feb (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.619451 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f12a4b397b8f4dedae851f98a8b18feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.621788 15055 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_88be21f10fc74ddaab680d288fbb9a13" } new_table_name: "default.Table_b0d6857034b54b258f3f6d577cd53e13" modify_external_catalogs: true
I20240501 19:57:26.625633 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 16789653df71436a9ba290bfc919cb25 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.626009 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16789653df71436a9ba290bfc919cb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.626194 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 16789653df71436a9ba290bfc919cb25 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.626470 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16789653df71436a9ba290bfc919cb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.628137 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 16789653df71436a9ba290bfc919cb25 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.628459 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16789653df71436a9ba290bfc919cb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.631598 15712 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.634317 15712 tablet_bootstrap.cc:654] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.635890 15712 log.cc:824] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:26.637823 15712 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.638012 15712 ts_tablet_manager.cc:1367] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 19:57:26.641220 15714 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.641598 15712 raft_consensus.cc:348] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.641978 15712 raft_consensus.cc:374] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.642076 15712 raft_consensus.cc:729] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.642670 15712 consensus_queue.cc:260] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.643468 15713 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.643751 15712 ts_tablet_manager.cc:1398] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:57:26.645458 15714 tablet_bootstrap.cc:654] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.646811 15712 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.647780 15713 tablet_bootstrap.cc:654] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.648229 15712 tablet_bootstrap.cc:654] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.648681 15713 log.cc:824] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:26.649525 15712 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.649657 15712 ts_tablet_manager.cc:1367] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:26.650283 15712 raft_consensus.cc:348] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.650461 15712 raft_consensus.cc:374] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.650535 15712 raft_consensus.cc:729] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.650758 15058 catalog_manager.cc:2145] created HMS catalog entry for table [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:26.651059 15712 consensus_queue.cc:260] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.651540 15712 ts_tablet_manager.cc:1398] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:26.651988 15714 log.cc:824] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:26.654531 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--webserver_interface=127.14.96.60
--webserver_port=43311
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:26.659209 15713 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.661837 15713 ts_tablet_manager.cc:1367] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20240501 19:57:26.664225 15306 tablet_service.cc:1467] Processing CreateTablet for tablet ad5586094d51448581f25db4db838305 (DEFAULT_TABLE table=default.table_39fb93a866e54177a673e26dd8325964 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.664521 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5586094d51448581f25db4db838305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.665501 15444 tablet_service.cc:1467] Processing CreateTablet for tablet ad5586094d51448581f25db4db838305 (DEFAULT_TABLE table=default.table_39fb93a866e54177a673e26dd8325964 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.665756 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5586094d51448581f25db4db838305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.667819 15714 tablet_bootstrap.cc:492] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.667855 15712 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.668046 15714 ts_tablet_manager.cc:1367] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240501 19:57:26.668931 15712 tablet_bootstrap.cc:654] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.669178 15058 master_service.cc:946] ReplaceTablet: received request to replace tablet f12a4b397b8f4dedae851f98a8b18feb from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:26.671500 15580 tablet_service.cc:1467] Processing CreateTablet for tablet ad5586094d51448581f25db4db838305 (DEFAULT_TABLE table=default.table_39fb93a866e54177a673e26dd8325964 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.671784 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5586094d51448581f25db4db838305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.671876 15714 raft_consensus.cc:348] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.672089 15714 raft_consensus.cc:374] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.672143 15714 raft_consensus.cc:729] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.672616 15714 consensus_queue.cc:260] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.675504 15713 raft_consensus.cc:348] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.675735 15713 raft_consensus.cc:374] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.675788 15713 raft_consensus.cc:729] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.676652 15712 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.676815 15712 ts_tablet_manager.cc:1367] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 19:57:26.677392 15712 raft_consensus.cc:348] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.677565 15712 raft_consensus.cc:374] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.677644 15712 raft_consensus.cc:729] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.677657 15714 ts_tablet_manager.cc:1398] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240501 19:57:26.677709 15058 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet f12a4b397b8f4dedae851f98a8b18feb
I20240501 19:57:26.677860 15712 consensus_queue.cc:260] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.678138 15058 catalog_manager.cc:6554] ReplaceTablet: tablet f12a4b397b8f4dedae851f98a8b18feb deleted and replaced by tablet 14b4c5836101423ab760756c51a8c72b
I20240501 19:57:26.678315 15712 ts_tablet_manager.cc:1398] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.679395 15058 master_service.cc:946] ReplaceTablet: received request to replace tablet 16789653df71436a9ba290bfc919cb25 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:26.680778 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:26.680910 15713 consensus_queue.cc:260] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.683352 15714 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.684998 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:26.685717 15713 ts_tablet_manager.cc:1398] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240501 19:57:26.685603 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:26.687074 15714 tablet_bootstrap.cc:654] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.689601 15713 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.690589 15713 tablet_bootstrap.cc:654] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.690912 15058 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 16789653df71436a9ba290bfc919cb25
I20240501 19:57:26.691237 15058 catalog_manager.cc:6554] ReplaceTablet: tablet 16789653df71436a9ba290bfc919cb25 deleted and replaced by tablet 84e74448c8994843bf8b2335669e3713
I20240501 19:57:26.692013 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 16789653df71436a9ba290bfc919cb25 with delete_type TABLET_DATA_DELETED (replaced by tablet 84e74448c8994843bf8b2335669e3713) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:26.693297 15058 master_service.cc:946] ReplaceTablet: received request to replace tablet 16789653df71436a9ba290bfc919cb25 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:26.693575 15058 master_service.cc:946] ReplaceTablet: received request to replace tablet f12a4b397b8f4dedae851f98a8b18feb from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:26.693835 15058 master_service.cc:946] ReplaceTablet: received request to replace tablet f12a4b397b8f4dedae851f98a8b18feb from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:26.694114 15058 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_39fb93a866e54177a673e26dd8325964" } new_table_name: "default.Table_0ef1c033257e4e358ccbe701cce981e2" modify_external_catalogs: true
I20240501 19:57:26.694293 15056 catalog_manager.cc:2145] created HMS catalog entry for table [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:26.696931 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet 16789653df71436a9ba290bfc919cb25 with delete_type TABLET_DATA_DELETED (replaced by tablet 84e74448c8994843bf8b2335669e3713) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:26.697482 15726 tablet_replica.cc:330] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
W20240501 19:57:26.697603 15048 catalog_manager.cc:4810] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): delete failed for tablet f12a4b397b8f4dedae851f98a8b18feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f12a4b397b8f4dedae851f98a8b18feb already in progress: creating tablet
I20240501 19:57:26.697794 15726 raft_consensus.cc:2229] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.698088 15726 raft_consensus.cc:2258] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.699003 15727 tablet_replica.cc:330] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
W20240501 19:57:26.699052 15050 catalog_manager.cc:4810] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): delete failed for tablet f12a4b397b8f4dedae851f98a8b18feb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f12a4b397b8f4dedae851f98a8b18feb already in progress: creating tablet
I20240501 19:57:26.699270 15727 raft_consensus.cc:2229] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.699524 15727 raft_consensus.cc:2258] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.700650 15713 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.700743 15713 ts_tablet_manager.cc:1367] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:26.701308 15713 raft_consensus.cc:348] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.701422 15713 raft_consensus.cc:374] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.701465 15713 raft_consensus.cc:729] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.701606 15713 consensus_queue.cc:260] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.701851 15713 ts_tablet_manager.cc:1398] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.701911 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 16789653df71436a9ba290bfc919cb25 with delete_type TABLET_DATA_DELETED (replaced by tablet 84e74448c8994843bf8b2335669e3713) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:26.702811 15056 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_a94d910a1db245728708bdc7eb0b50b2" } new_table_name: "default.Table_0441f7d080e84fb792251357a57aa031" modify_external_catalogs: true
I20240501 19:57:26.702873 15724 tablet_replica.cc:330] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:26.703245 15724 raft_consensus.cc:2229] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.703370 15714 tablet_bootstrap.cc:492] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.703518 15714 ts_tablet_manager.cc:1367] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:57:26.703752 15724 raft_consensus.cc:2258] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.704241 15714 raft_consensus.cc:348] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.704414 15714 raft_consensus.cc:374] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.704497 15714 raft_consensus.cc:729] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.704764 15714 consensus_queue.cc:260] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.705017 15724 ts_tablet_manager.cc:1875] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.705163 15714 ts_tablet_manager.cc:1398] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:26.706076 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4c5836101423ab760756c51a8c72b (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.706365 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4c5836101423ab760756c51a8c72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.707147 15714 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.708148 15714 tablet_bootstrap.cc:654] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.709251 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4c5836101423ab760756c51a8c72b (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.709475 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4c5836101423ab760756c51a8c72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.709605 15727 ts_tablet_manager.cc:1875] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.712081 15726 ts_tablet_manager.cc:1875] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.713692 15713 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.714553 15713 tablet_bootstrap.cc:654] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.715117 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 84e74448c8994843bf8b2335669e3713 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.716508 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e74448c8994843bf8b2335669e3713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.718906 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 84e74448c8994843bf8b2335669e3713 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.719180 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e74448c8994843bf8b2335669e3713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.719260 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f (DEFAULT_TABLE table=default.table_a94d910a1db245728708bdc7eb0b50b2 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.719470 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d695b92bd1a4188b600a0dadd04cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.722360 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f (DEFAULT_TABLE table=default.table_a94d910a1db245728708bdc7eb0b50b2 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.722602 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d695b92bd1a4188b600a0dadd04cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.722810 15727 ts_tablet_manager.cc:1888] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.723359 15726 ts_tablet_manager.cc:1888] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.723542 15727 log.cc:1196] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/16789653df71436a9ba290bfc919cb25
I20240501 19:57:26.723924 15727 ts_tablet_manager.cc:1909] T 16789653df71436a9ba290bfc919cb25 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:26.724046 15726 log.cc:1196] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/16789653df71436a9ba290bfc919cb25
I20240501 19:57:26.725373 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 16789653df71436a9ba290bfc919cb25 (table default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:26.725800 15726 ts_tablet_manager.cc:1909] T 16789653df71436a9ba290bfc919cb25 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:26.708734 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4c5836101423ab760756c51a8c72b (DEFAULT_TABLE table=default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.729204 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4c5836101423ab760756c51a8c72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.713069 15443 tablet_service.cc:1467] Processing CreateTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f (DEFAULT_TABLE table=default.table_a94d910a1db245728708bdc7eb0b50b2 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.731472 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d695b92bd1a4188b600a0dadd04cc7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.731748 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 16789653df71436a9ba290bfc919cb25 (table default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:26.714358 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 84e74448c8994843bf8b2335669e3713 (DEFAULT_TABLE table=default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.733799 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84e74448c8994843bf8b2335669e3713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.736649 15712 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.737859 15712 tablet_bootstrap.cc:654] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.742170 15712 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.742370 15712 ts_tablet_manager.cc:1367] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:26.742959 15712 raft_consensus.cc:348] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.743124 15712 raft_consensus.cc:374] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.743166 15712 raft_consensus.cc:729] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.743379 15712 consensus_queue.cc:260] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.743687 15712 ts_tablet_manager.cc:1398] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.743860 15712 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.744809 15712 tablet_bootstrap.cc:654] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.745039 15059 catalog_manager.cc:2145] created HMS catalog entry for table [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:26.751489 15724 ts_tablet_manager.cc:1888] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.751695 15724 log.cc:1196] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/f12a4b397b8f4dedae851f98a8b18feb
I20240501 19:57:26.752012 15724 ts_tablet_manager.cc:1909] T f12a4b397b8f4dedae851f98a8b18feb P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:26.753612 15714 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.753731 15714 ts_tablet_manager.cc:1367] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.000s
I20240501 19:57:26.754220 15714 raft_consensus.cc:348] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.754335 15714 raft_consensus.cc:374] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.754362 15714 raft_consensus.cc:729] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.754485 15714 consensus_queue.cc:260] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.754760 15714 ts_tablet_manager.cc:1398] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.755388 15714 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.756326 15714 tablet_bootstrap.cc:654] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.761909 15724 tablet_replica.cc:330] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:26.762073 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet f12a4b397b8f4dedae851f98a8b18feb (table default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:26.762526 15713 tablet_bootstrap.cc:492] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.762634 15713 ts_tablet_manager.cc:1367] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.002s
I20240501 19:57:26.763098 15713 raft_consensus.cc:348] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.763224 15713 raft_consensus.cc:374] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.763253 15713 raft_consensus.cc:729] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.763373 15713 consensus_queue.cc:260] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.763633 15713 ts_tablet_manager.cc:1398] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.763824 15713 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.766937 15579 tablet_service.cc:1467] Processing CreateTablet for tablet f7c20396c4be498ebb07eb388103afdc (DEFAULT_TABLE table=default.table_934e8fcbba9a4c2a9e875db16c2ff080 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.767253 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c20396c4be498ebb07eb388103afdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.764820 15713 tablet_bootstrap.cc:654] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.766350 15305 tablet_service.cc:1467] Processing CreateTablet for tablet f7c20396c4be498ebb07eb388103afdc (DEFAULT_TABLE table=default.table_934e8fcbba9a4c2a9e875db16c2ff080 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.768329 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c20396c4be498ebb07eb388103afdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.770067 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:26.770272 15727 tablet_replica.cc:330] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:26.770335 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:26.770419 15727 raft_consensus.cc:2229] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.770552 15727 raft_consensus.cc:2258] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.770617 15059 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_934e8fcbba9a4c2a9e875db16c2ff080" } new_table_name: "default.Table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5" modify_external_catalogs: true
I20240501 19:57:26.770705 15726 tablet_replica.cc:330] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:26.770803 15726 raft_consensus.cc:2229] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.770895 15726 raft_consensus.cc:2258] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.771633 15726 ts_tablet_manager.cc:1875] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.767057 15444 tablet_service.cc:1467] Processing CreateTablet for tablet f7c20396c4be498ebb07eb388103afdc (DEFAULT_TABLE table=default.table_934e8fcbba9a4c2a9e875db16c2ff080 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.773113 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7c20396c4be498ebb07eb388103afdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.773705 15726 ts_tablet_manager.cc:1888] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.773768 15726 log.cc:1196] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/f12a4b397b8f4dedae851f98a8b18feb
I20240501 19:57:26.773962 15726 ts_tablet_manager.cc:1909] T f12a4b397b8f4dedae851f98a8b18feb P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:26.774645 15727 ts_tablet_manager.cc:1875] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.775833 15724 raft_consensus.cc:2229] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:26.775972 15724 raft_consensus.cc:2258] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:26.776723 15714 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.776821 15714 ts_tablet_manager.cc:1367] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.002s
I20240501 19:57:26.777041 15727 ts_tablet_manager.cc:1888] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.777098 15727 log.cc:1196] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/f12a4b397b8f4dedae851f98a8b18feb
I20240501 19:57:26.777310 15727 ts_tablet_manager.cc:1909] T f12a4b397b8f4dedae851f98a8b18feb P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:26.777412 15714 raft_consensus.cc:348] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.777544 15714 raft_consensus.cc:374] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.777580 15714 raft_consensus.cc:729] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.777756 15714 consensus_queue.cc:260] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.778034 15714 ts_tablet_manager.cc:1398] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.778259 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet f12a4b397b8f4dedae851f98a8b18feb (table default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:26.779023 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet f12a4b397b8f4dedae851f98a8b18feb (table default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:26.780121 15714 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.781142 15714 tablet_bootstrap.cc:654] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.782900 15724 ts_tablet_manager.cc:1875] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:26.785053 15724 ts_tablet_manager.cc:1888] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:26.785136 15724 log.cc:1196] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/16789653df71436a9ba290bfc919cb25
I20240501 19:57:26.785353 15724 ts_tablet_manager.cc:1909] T 16789653df71436a9ba290bfc919cb25 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:26.786092 15713 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.786214 15713 ts_tablet_manager.cc:1367] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.002s
I20240501 19:57:26.786685 15713 raft_consensus.cc:348] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.786811 15713 raft_consensus.cc:374] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.786851 15713 raft_consensus.cc:729] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.786993 15713 consensus_queue.cc:260] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.787257 15713 ts_tablet_manager.cc:1398] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.787449 15713 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.790665 15713 tablet_bootstrap.cc:654] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.793294 15719 raft_consensus.cc:482] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.793474 15719 raft_consensus.cc:504] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.794675 15719 leader_election.cc:288] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.801357 15719 raft_consensus.cc:482] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.801487 15719 raft_consensus.cc:504] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.801544 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:26.801921 15719 leader_election.cc:288] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
W20240501 19:57:26.802537 15397 leader_election.cc:341] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:26.802983 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5586094d51448581f25db4db838305" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:26.803216 15326 raft_consensus.cc:2454] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:26.803644 15397 leader_election.cc:302] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:26.803999 15719 raft_consensus.cc:2781] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:26.804070 15719 raft_consensus.cc:482] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:26.804111 15719 raft_consensus.cc:3037] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.805757 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:26.805907 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:26.808674 15719 raft_consensus.cc:504] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.809353 15719 leader_election.cc:288] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.809814 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5586094d51448581f25db4db838305" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:26.809947 15326 raft_consensus.cc:3037] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.811043 15326 raft_consensus.cc:2454] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:26.811479 15057 catalog_manager.cc:2145] created HMS catalog entry for table [id=0ef7499159164fcf9c5b0d5e4484153c]
I20240501 19:57:26.811436 15397 leader_election.cc:302] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:26.813421 15716 raft_consensus.cc:2781] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:26.813664 15716 raft_consensus.cc:686] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:26.813918 15716 consensus_queue.cc:237] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.819149 15057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_a63257f60f4e4b9ca1d285937e0fe0e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:57:26.820693 15649 tablet.cc:2369] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:26.821995 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet f12a4b397b8f4dedae851f98a8b18feb with delete_type TABLET_DATA_DELETED (replaced by tablet 14b4c5836101423ab760756c51a8c72b) from {username='slave'} at 127.0.0.1:40964
W20240501 19:57:26.822801 15048 catalog_manager.cc:4797] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): delete failed for tablet f12a4b397b8f4dedae851f98a8b18feb because the tablet was not found. No further retry: Not found: Tablet not found: f12a4b397b8f4dedae851f98a8b18feb
I20240501 19:57:26.830878 15600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:26.830924 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (DEFAULT_TABLE table=default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.831219 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.831501 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5586094d51448581f25db4db838305" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:26.831629 15598 raft_consensus.cc:3037] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.832726 15598 raft_consensus.cc:2454] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:26.832937 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5586094d51448581f25db4db838305" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:26.833077 15599 raft_consensus.cc:2362] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1. Re-sending same reply.
I20240501 19:57:26.833791 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (DEFAULT_TABLE table=default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.833803 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (DEFAULT_TABLE table=default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.834056 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.834072 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:57:26.836797 15399 leader_election.cc:341] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:26.836920 15399 leader_election.cc:302] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23; no voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:26.838761 15713 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.838936 15713 ts_tablet_manager.cc:1367] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.002s
I20240501 19:57:26.839668 15713 raft_consensus.cc:348] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.839869 15713 raft_consensus.cc:374] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.839942 15713 raft_consensus.cc:729] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.840152 15713 consensus_queue.cc:260] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.840502 15713 ts_tablet_manager.cc:1398] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.843322 15713 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.844408 15713 tablet_bootstrap.cc:654] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.846698 15719 raft_consensus.cc:2726] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:26.848393 15712 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.848511 15712 ts_tablet_manager.cc:1367] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.000s
I20240501 19:57:26.848572 15714 tablet_bootstrap.cc:492] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.848663 15714 ts_tablet_manager.cc:1367] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.069s user 0.000s sys 0.002s
I20240501 19:57:26.849045 15712 raft_consensus.cc:348] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.849159 15712 raft_consensus.cc:374] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.849191 15712 raft_consensus.cc:729] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.849263 15714 raft_consensus.cc:348] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.849321 15712 consensus_queue.cc:260] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.849372 15714 raft_consensus.cc:374] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.849406 15714 raft_consensus.cc:729] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.849602 15712 ts_tablet_manager.cc:1398] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.849576 15714 consensus_queue.cc:260] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.849831 15712 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.849838 15714 ts_tablet_manager.cc:1398] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.850761 15712 tablet_bootstrap.cc:654] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.850792 15714 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.851913 15714 tablet_bootstrap.cc:654] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.855824 15712 tablet_bootstrap.cc:492] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.856009 15712 ts_tablet_manager.cc:1367] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:26.856577 15712 raft_consensus.cc:348] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.856762 15712 raft_consensus.cc:374] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.856861 15712 raft_consensus.cc:729] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.857074 15712 consensus_queue.cc:260] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.857434 15712 ts_tablet_manager.cc:1398] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:26.858276 15712 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.859284 15714 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.859391 15714 ts_tablet_manager.cc:1367] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:57:26.859408 15712 tablet_bootstrap.cc:654] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.860337 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 16789653df71436a9ba290bfc919cb25 (table default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:26.860481 15714 raft_consensus.cc:348] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.860586 15714 raft_consensus.cc:374] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.860617 15714 raft_consensus.cc:729] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.860757 15714 consensus_queue.cc:260] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.861130 15714 ts_tablet_manager.cc:1398] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:26.861339 15714 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.862298 15714 tablet_bootstrap.cc:654] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.863130 15723 raft_consensus.cc:482] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.863229 15723 raft_consensus.cc:504] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.864492 15723 leader_election.cc:288] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:26.865512 15712 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.865610 15712 ts_tablet_manager.cc:1367] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:57:26.866079 15712 raft_consensus.cc:348] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.866209 15712 raft_consensus.cc:374] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.866245 15712 raft_consensus.cc:729] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.866319 15063 catalog_manager.cc:2145] created HMS catalog entry for table [id=a8f768afdece4f33a1089d673dc188ec]
I20240501 19:57:26.866514 15712 consensus_queue.cc:260] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.868247 15712 ts_tablet_manager.cc:1398] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:26.869108 15712 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
W20240501 19:57:26.869985 15062 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0ca797a676d54ea89cfea3963a75b1d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:26.870290 15712 tablet_bootstrap.cc:654] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:26.870743 15064 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0c03b0e181d140c88754b558e4fb4393 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:26.871356 15712 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
W20240501 19:57:26.871356 15061 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_31c7944e4b41499683c266b366d0711f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:26.871426 15712 ts_tablet_manager.cc:1367] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:26.871883 15712 raft_consensus.cc:348] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.872047 15712 raft_consensus.cc:374] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.872117 15712 raft_consensus.cc:729] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.872306 15712 consensus_queue.cc:260] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
W20240501 19:57:26.874236 15513 tablet.cc:2369] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:26.874099 15712 ts_tablet_manager.cc:1398] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:26.878660 15719 raft_consensus.cc:482] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.878767 15719 raft_consensus.cc:504] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.879129 15719 leader_election.cc:288] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:26.880123 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:26.880635 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
W20240501 19:57:26.881088 15397 leader_election.cc:341] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:57:26.881750 15399 leader_election.cc:341] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:26.881821 15399 leader_election.cc:302] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23; no voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:26.882010 15719 raft_consensus.cc:2726] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:26.886730 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (DEFAULT_TABLE table=default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.887066 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.887996 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:26.888126 15326 raft_consensus.cc:2454] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:26.888566 15533 leader_election.cc:302] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:26.890504 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (DEFAULT_TABLE table=default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.890326 15063 catalog_manager.cc:5484] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:26.890767 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.893659 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (DEFAULT_TABLE table=default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.893949 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.897758 15712 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.898732 15712 tablet_bootstrap.cc:654] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.902289 15723 raft_consensus.cc:2781] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:26.902376 15723 raft_consensus.cc:482] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:26.902411 15723 raft_consensus.cc:3037] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.903426 15712 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:26.903429 15723 raft_consensus.cc:504] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.903515 15712 ts_tablet_manager.cc:1367] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:26.903960 15712 raft_consensus.cc:348] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.904072 15712 raft_consensus.cc:374] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.904099 15712 raft_consensus.cc:729] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.904215 15712 consensus_queue.cc:260] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.904364 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:26.904453 15712 ts_tablet_manager.cc:1398] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.904484 15326 raft_consensus.cc:3037] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.905570 15326 raft_consensus.cc:2454] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:26.907915 15533 leader_election.cc:302] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:26.908412 15723 leader_election.cc:288] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:26.908552 15723 raft_consensus.cc:2781] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:26.908653 15723 raft_consensus.cc:686] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:26.908962 15723 consensus_queue.cc:237] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.915305 15063 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_40db4cea1aad411796b548b57c595842" } modify_external_catalogs: true
I20240501 19:57:26.915481 15063 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_40db4cea1aad411796b548b57c595842" } modify_external_catalogs: true
I20240501 19:57:26.916800 15719 raft_consensus.cc:482] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.916918 15719 raft_consensus.cc:504] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:26.917356 15719 leader_election.cc:288] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.917672 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:26.917881 15326 raft_consensus.cc:2379] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68539c0ecd3e44e080cb53987d4f4c23 in current term 1: Already voted for candidate 7f011d422e164c949a9d54d846b7c432 in this term.
I20240501 19:57:26.919292 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:26.919607 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:26.919800 15464 raft_consensus.cc:2454] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:26.919847 15399 leader_election.cc:302] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23; no voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:26.920063 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84e74448c8994843bf8b2335669e3713" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
W20240501 19:57:26.920393 15057 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:26.920076 15719 raft_consensus.cc:3037] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.921989 15719 raft_consensus.cc:2726] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:26.923409 15463 raft_consensus.cc:2454] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
W20240501 19:57:26.926596 15375 tablet.cc:2369] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:26.929503 15713 tablet_bootstrap.cc:492] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.929704 15713 ts_tablet_manager.cc:1367] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.086s user 0.001s sys 0.001s
I20240501 19:57:26.930295 15713 raft_consensus.cc:348] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.930516 15713 raft_consensus.cc:374] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.930606 15713 raft_consensus.cc:729] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.930828 15713 consensus_queue.cc:260] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.932477 15714 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.932580 15714 ts_tablet_manager.cc:1367] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.000s
I20240501 19:57:26.933072 15714 raft_consensus.cc:348] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.933177 15714 raft_consensus.cc:374] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.933220 15714 raft_consensus.cc:729] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.933357 15714 consensus_queue.cc:260] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.933622 15714 ts_tablet_manager.cc:1398] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.933797 15714 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.934964 15714 tablet_bootstrap.cc:654] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.937440 15714 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.937536 15714 ts_tablet_manager.cc:1367] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:26.937947 15714 raft_consensus.cc:348] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.938041 15714 raft_consensus.cc:374] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.938079 15714 raft_consensus.cc:729] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.938254 15714 consensus_queue.cc:260] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.938531 15714 ts_tablet_manager.cc:1398] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.941146 15714 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.942073 15714 tablet_bootstrap.cc:654] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.942917 15713 ts_tablet_manager.cc:1398] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 19:57:26.943370 15714 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.943454 15714 ts_tablet_manager.cc:1367] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:26.943884 15714 raft_consensus.cc:348] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.943974 15714 raft_consensus.cc:374] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.944008 15714 raft_consensus.cc:729] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.944214 15714 consensus_queue.cc:260] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.946027 15714 ts_tablet_manager.cc:1398] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:26.947417 15713 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.948555 15713 tablet_bootstrap.cc:654] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.950619 15713 tablet_bootstrap.cc:492] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.950805 15713 ts_tablet_manager.cc:1367] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:26.951270 15713 raft_consensus.cc:348] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.951366 15713 raft_consensus.cc:374] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.951400 15713 raft_consensus.cc:729] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.951565 15713 consensus_queue.cc:260] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.951848 15713 ts_tablet_manager.cc:1398] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.952030 15713 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.953071 15713 tablet_bootstrap.cc:654] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:26.954358 15720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:26.954735 15720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:26.954843 15720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 19:57:26.957578 15713 tablet_bootstrap.cc:492] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.957690 15713 ts_tablet_manager.cc:1367] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:26.958215 15713 raft_consensus.cc:348] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.958343 15713 raft_consensus.cc:374] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.958386 15713 raft_consensus.cc:729] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.958514 15713 consensus_queue.cc:260] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.958842 15713 ts_tablet_manager.cc:1398] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.959055 15713 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
W20240501 19:57:26.959476 15720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:26.959534 15720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:26.959568 15720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:26.959596 15720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:26.960255 15713 tablet_bootstrap.cc:654] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.964426 15720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.96.60
--webserver_port=43311
--never_fsync=true
--heap_profile_path=/tmp/kudu.15720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:26.965684 15720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:26.967177 15720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:26.968106 15713 tablet_bootstrap.cc:492] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.968312 15713 ts_tablet_manager.cc:1367] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:26.968555 15062 catalog_manager.cc:2145] created HMS catalog entry for table [id=f95181efc2104a7ab9796cc0e11a64e4]
I20240501 19:57:26.974223 15713 raft_consensus.cc:348] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.974576 15713 raft_consensus.cc:374] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.974692 15713 raft_consensus.cc:729] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.974854 15713 consensus_queue.cc:260] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.976013 15713 ts_tablet_manager.cc:1398] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:26.980094 15062 catalog_manager.cc:5484] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:26.981789 15444 tablet_service.cc:1467] Processing CreateTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded (DEFAULT_TABLE table=default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.982075 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79d98a2beff4dc690b39a0a5ac04ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.983139 15580 tablet_service.cc:1467] Processing CreateTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded (DEFAULT_TABLE table=default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.983455 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79d98a2beff4dc690b39a0a5ac04ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.983533 15719 raft_consensus.cc:482] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:26.983613 15719 raft_consensus.cc:504] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.983974 15719 leader_election.cc:288] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.984179 15306 tablet_service.cc:1467] Processing CreateTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded (DEFAULT_TABLE table=default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:26.984431 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e79d98a2beff4dc690b39a0a5ac04ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:26.984372 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2fe4a03e9b4fe9ac80b3c95b5bbc8e" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:26.984508 15598 raft_consensus.cc:2454] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:26.984802 15399 leader_election.cc:302] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:26.984982 15719 raft_consensus.cc:2781] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:26.985020 15719 raft_consensus.cc:482] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:26.985054 15719 raft_consensus.cc:3037] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.985900 15719 raft_consensus.cc:504] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.986289 15719 leader_election.cc:288] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:26.986536 15713 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:26.986853 15712 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:26.987917 15712 tablet_bootstrap.cc:654] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.988108 15713 tablet_bootstrap.cc:654] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.988638 15714 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:26.988946 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2fe4a03e9b4fe9ac80b3c95b5bbc8e" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:26.989068 15598 raft_consensus.cc:3037] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.989578 15714 tablet_bootstrap.cc:654] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:26.990137 15598 raft_consensus.cc:2454] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:26.990511 15399 leader_election.cc:302] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:26.990696 15719 raft_consensus.cc:2781] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:26.990726 15714 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:26.990769 15719 raft_consensus.cc:686] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:26.990818 15714 ts_tablet_manager.cc:1367] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:26.990929 15719 consensus_queue.cc:237] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.991259 15714 raft_consensus.cc:348] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.991359 15714 raft_consensus.cc:374] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.991389 15714 raft_consensus.cc:729] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.991509 15714 consensus_queue.cc:260] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.991767 15714 ts_tablet_manager.cc:1398] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:26.994239 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2fe4a03e9b4fe9ac80b3c95b5bbc8e" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:26.994403 15326 raft_consensus.cc:2454] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:26.994644 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f2fe4a03e9b4fe9ac80b3c95b5bbc8e" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:26.994750 15325 raft_consensus.cc:3037] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:26.995841 15325 raft_consensus.cc:2454] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:26.997917 15713 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:26.998014 15713 ts_tablet_manager.cc:1367] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20240501 19:57:26.998528 15713 raft_consensus.cc:348] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.998628 15713 raft_consensus.cc:374] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:26.998659 15713 raft_consensus.cc:729] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:26.998806 15713 consensus_queue.cc:260] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:26.999107 15713 ts_tablet_manager.cc:1398] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:26.999889 15062 catalog_manager.cc:5484] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:27.004066 15712 tablet_bootstrap.cc:492] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:27.004179 15712 ts_tablet_manager.cc:1367] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240501 19:57:27.004623 15712 raft_consensus.cc:348] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.004765 15712 raft_consensus.cc:374] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.004801 15712 raft_consensus.cc:729] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
W20240501 19:57:27.005112 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:27.006198 15752 raft_consensus.cc:482] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.006309 15752 raft_consensus.cc:504] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.006764 15752 leader_election.cc:288] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.007323 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47dc8f2e6b4e4e9b8d8a7b6f140e34" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.007449 15463 raft_consensus.cc:2454] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
W20240501 19:57:27.007803 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:27.007930 15536 leader_election.cc:302] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.008488 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47dc8f2e6b4e4e9b8d8a7b6f140e34" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:27.008647 15325 raft_consensus.cc:2454] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.009572 15752 raft_consensus.cc:2781] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.009641 15752 raft_consensus.cc:482] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.009680 15752 raft_consensus.cc:3037] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.010512 15752 raft_consensus.cc:504] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.010831 15752 leader_election.cc:288] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.011214 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47dc8f2e6b4e4e9b8d8a7b6f140e34" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.011262 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e47dc8f2e6b4e4e9b8d8a7b6f140e34" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:27.011368 15463 raft_consensus.cc:3037] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.011411 15325 raft_consensus.cc:3037] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.012423 15325 raft_consensus.cc:2454] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.012442 15463 raft_consensus.cc:2454] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.012768 15533 leader_election.cc:302] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.013234 15752 raft_consensus.cc:2781] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.013320 15752 raft_consensus.cc:686] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:27.013461 15752 consensus_queue.cc:237] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.014786 15712 consensus_queue.cc:260] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.015137 15712 ts_tablet_manager.cc:1398] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:57:27.019515 15062 catalog_manager.cc:5484] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:27.020200 15064 catalog_manager.cc:2145] created HMS catalog entry for table [id=2434824a54f245fba1c41a6edcc62dad]
W20240501 19:57:27.021976 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:27.024358 15062 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_ea70d92de3fc40b4b3e8c54d6e5b6899"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:27.026335 15064 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:27.026954 15720 server_base.cc:1034] running on GCE node
I20240501 19:57:27.027659 15720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:27.029516 15720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:27.031204 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.031520 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f46ab4e72478ca29bfca14391db07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.032011 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.032271 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f46ab4e72478ca29bfca14391db07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.034593 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.034674 15713 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:27.034848 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e3f46ab4e72478ca29bfca14391db07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.035669 15713 tablet_bootstrap.cc:654] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.036506 15714 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:27.037071 15712 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:27.038270 15714 tablet_bootstrap.cc:654] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.038506 15712 tablet_bootstrap.cc:654] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.039397 15714 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:27.039485 15714 ts_tablet_manager.cc:1367] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 19:57:27.039709 15713 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:27.039785 15713 ts_tablet_manager.cc:1367] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:57:27.039963 15714 raft_consensus.cc:348] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.040074 15714 raft_consensus.cc:374] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.040107 15714 raft_consensus.cc:729] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.040282 15713 raft_consensus.cc:348] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.040398 15713 raft_consensus.cc:374] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.040448 15713 raft_consensus.cc:729] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.040577 15713 consensus_queue.cc:260] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.040561 15714 consensus_queue.cc:260] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.040894 15713 ts_tablet_manager.cc:1398] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:27.042294 15714 ts_tablet_manager.cc:1398] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:27.048331 15712 tablet_bootstrap.cc:492] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:27.048493 15712 ts_tablet_manager.cc:1367] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:27.049113 15712 raft_consensus.cc:348] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.049257 15712 raft_consensus.cc:374] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.049288 15712 raft_consensus.cc:729] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.049443 15712 consensus_queue.cc:260] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.049746 15712 ts_tablet_manager.cc:1398] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:27.051251 15720 hybrid_clock.cc:648] HybridClock initialized: now 1714593447051191 us; error 48 us; skew 500 ppm
I20240501 19:57:27.053887 15720 webserver.cc:460] Webserver started at http://127.14.96.60:43311/ using document root <none> and password file <none>
I20240501 19:57:27.054613 15720 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:27.054706 15720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:27.057195 15061 catalog_manager.cc:2145] created HMS catalog entry for table [id=076667aca4134cf9a588a3dc30297332]
I20240501 19:57:27.059326 15720 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:57:27.067689 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:27.068977 15752 raft_consensus.cc:482] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.069075 15752 raft_consensus.cc:504] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.069447 15752 leader_election.cc:288] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.069904 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79d98a2beff4dc690b39a0a5ac04ded" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:27.069970 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 4814dbbccd08455aa0b7381de1a34616 (DEFAULT_TABLE table=default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.070044 15325 raft_consensus.cc:2454] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.070286 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4814dbbccd08455aa0b7381de1a34616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.070514 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79d98a2beff4dc690b39a0a5ac04ded" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.070652 15463 raft_consensus.cc:2454] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.070923 15536 leader_election.cc:302] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.072806 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 4814dbbccd08455aa0b7381de1a34616 (DEFAULT_TABLE table=default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.073133 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4814dbbccd08455aa0b7381de1a34616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.073834 15752 raft_consensus.cc:2781] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.073902 15752 raft_consensus.cc:482] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.073935 15752 raft_consensus.cc:3037] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.074882 15752 raft_consensus.cc:504] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.075238 15752 leader_election.cc:288] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.075515 15713 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:27.075695 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79d98a2beff4dc690b39a0a5ac04ded" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.075799 15463 raft_consensus.cc:3037] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.076571 15713 tablet_bootstrap.cc:654] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.076656 15463 raft_consensus.cc:2454] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.077256 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e79d98a2beff4dc690b39a0a5ac04ded" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:27.077349 15712 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:27.077389 15325 raft_consensus.cc:3037] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.078243 15325 raft_consensus.cc:2454] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.078644 15536 leader_election.cc:302] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.078884 15712 tablet_bootstrap.cc:654] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.080782 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 4814dbbccd08455aa0b7381de1a34616 (DEFAULT_TABLE table=default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.081128 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4814dbbccd08455aa0b7381de1a34616. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.081534 15713 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:27.081634 15713 ts_tablet_manager.cc:1367] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:27.082130 15713 raft_consensus.cc:348] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.082269 15713 raft_consensus.cc:374] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.082315 15713 raft_consensus.cc:729] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.082458 15713 consensus_queue.cc:260] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.083423 15713 ts_tablet_manager.cc:1398] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:27.084214 15714 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:27.085126 15714 tablet_bootstrap.cc:654] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.089768 15061 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_b78cbc9abde043b6a52288415772661c" } modify_external_catalogs: true
I20240501 19:57:27.089928 15061 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_b78cbc9abde043b6a52288415772661c" } modify_external_catalogs: true
I20240501 19:57:27.090499 15752 raft_consensus.cc:2781] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.090600 15752 raft_consensus.cc:686] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:27.090742 15752 consensus_queue.cc:237] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.094657 15714 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:27.094765 15714 ts_tablet_manager.cc:1367] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:27.095064 15720 fs_manager.cc:725] Time spent opening block manager: real 0.034s user 0.003s sys 0.000s
I20240501 19:57:27.095221 15712 tablet_bootstrap.cc:492] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:27.095305 15712 ts_tablet_manager.cc:1367] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 19:57:27.095240 15714 raft_consensus.cc:348] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.095346 15714 raft_consensus.cc:374] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.095377 15714 raft_consensus.cc:729] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.095497 15714 consensus_queue.cc:260] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.095219 15720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:27.095782 15714 ts_tablet_manager.cc:1398] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:27.095759 15712 raft_consensus.cc:348] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.095890 15712 raft_consensus.cc:374] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.095922 15712 raft_consensus.cc:729] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.095835 15720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:27.096040 15712 consensus_queue.cc:260] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.096320 15712 ts_tablet_manager.cc:1398] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:27.097911 15723 raft_consensus.cc:482] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.097997 15723 raft_consensus.cc:504] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.098398 15723 leader_election.cc:288] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.099035 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:27.099174 15325 raft_consensus.cc:2454] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.099426 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.099545 15463 raft_consensus.cc:2454] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.100100 15536 leader_election.cc:302] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.101652 15752 raft_consensus.cc:2781] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.101718 15752 raft_consensus.cc:482] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.101769 15752 raft_consensus.cc:3037] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.102772 15752 raft_consensus.cc:504] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.103133 15752 leader_election.cc:288] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.103480 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:27.103613 15325 raft_consensus.cc:3037] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.103642 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7c20396c4be498ebb07eb388103afdc" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.103755 15463 raft_consensus.cc:3037] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.104646 15463 raft_consensus.cc:2454] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.104661 15325 raft_consensus.cc:2454] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.105003 15536 leader_election.cc:302] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.105212 15723 raft_consensus.cc:482] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.105279 15723 raft_consensus.cc:504] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.105587 15723 leader_election.cc:288] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.105666 15723 raft_consensus.cc:2781] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.105717 15723 raft_consensus.cc:686] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:27.105832 15723 consensus_queue.cc:237] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.108073 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4c5836101423ab760756c51a8c72b" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.108196 15463 raft_consensus.cc:2454] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.108455 15536 leader_election.cc:302] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.108664 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4c5836101423ab760756c51a8c72b" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:27.108772 15325 raft_consensus.cc:2454] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.109140 15723 raft_consensus.cc:2781] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.109196 15723 raft_consensus.cc:482] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.109230 15723 raft_consensus.cc:3037] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.110214 15723 raft_consensus.cc:504] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.110531 15723 leader_election.cc:288] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.111105 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4c5836101423ab760756c51a8c72b" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.111215 15463 raft_consensus.cc:3037] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.112040 15463 raft_consensus.cc:2454] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.112319 15536 leader_election.cc:302] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.112373 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4c5836101423ab760756c51a8c72b" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:27.112493 15325 raft_consensus.cc:3037] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.112591 15723 raft_consensus.cc:2781] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.112696 15723 raft_consensus.cc:686] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:27.112915 15723 consensus_queue.cc:237] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.113348 15325 raft_consensus.cc:2454] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.117354 15720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:27.118141 15720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:27.118310 15720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:27.127231 15720 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.60:40075
I20240501 19:57:27.128692 15720 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
I20240501 19:57:27.136998 15840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.60:40075 every 8 connection(s)
I20240501 19:57:27.138691 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 15720
I20240501 19:57:27.140445 15841 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:27.142994 15841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap starting.
I20240501 19:57:27.149910 15841 log.cc:824] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:27.162477 15841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:27.162940 15841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap complete.
I20240501 19:57:27.167024 15725 raft_consensus.cc:482] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.167172 15725 raft_consensus.cc:504] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.167176 15841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:27.167619 15841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.168201 15841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:27.168473 15725 leader_election.cc:288] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:27.169770 15841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:27.175055 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:27.175218 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:27.175606 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814dbbccd08455aa0b7381de1a34616" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:27.175769 15598 raft_consensus.cc:2454] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:27.176548 15261 leader_election.cc:302] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.176970 15725 raft_consensus.cc:2781] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.177039 15725 raft_consensus.cc:482] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.177078 15725 raft_consensus.cc:3037] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.178213 15725 raft_consensus.cc:504] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.178627 15725 leader_election.cc:288] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:27.179742 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814dbbccd08455aa0b7381de1a34616" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:27.179876 15598 raft_consensus.cc:3037] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.182160 15598 raft_consensus.cc:2454] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:27.182621 15261 leader_election.cc:302] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.185410 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814dbbccd08455aa0b7381de1a34616" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.185592 15463 raft_consensus.cc:2454] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:27.185845 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4814dbbccd08455aa0b7381de1a34616" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.185953 15464 raft_consensus.cc:3037] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.187073 15464 raft_consensus.cc:2454] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:27.187942 15725 raft_consensus.cc:2781] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.188472 15725 raft_consensus.cc:686] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:27.188802 15860 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:27.188884 15860 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: loading cluster ID for follower catalog manager: success
I20240501 19:57:27.188804 15725 consensus_queue.cc:237] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.191385 15860 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: acquiring CA information for follower catalog manager: success
I20240501 19:57:27.194999 15860 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:27.197633 15372 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:27.200170 15796 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:50879; Asking this server to re-register.
I20240501 19:57:27.200876 15372 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:27.202528 15796 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:27.208679 15865 raft_consensus.cc:482] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.208981 15865 raft_consensus.cc:504] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.209975 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:27.210175 15598 raft_consensus.cc:2454] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:27.210659 15261 leader_election.cc:302] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.210845 15725 raft_consensus.cc:2781] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.210896 15725 raft_consensus.cc:482] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.210947 15725 raft_consensus.cc:3037] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.212031 15725 raft_consensus.cc:504] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.212450 15725 leader_election.cc:288] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.213145 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.213287 15464 raft_consensus.cc:3037] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.214283 15464 raft_consensus.cc:2454] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:27.215103 15262 leader_election.cc:302] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:27.215629 15725 raft_consensus.cc:2781] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.215853 15725 raft_consensus.cc:686] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:27.216023 15725 consensus_queue.cc:237] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:27.217031 15865 leader_election.cc:288] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.217357 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:27.217501 15598 raft_consensus.cc:3037] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.218600 15598 raft_consensus.cc:2454] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:27.219383 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d695b92bd1a4188b600a0dadd04cc7f" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.219527 15464 raft_consensus.cc:2362] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1. Re-sending same reply.
I20240501 19:57:27.264952 15719 consensus_queue.cc:1035] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.273211 15646 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:27.275002 15796 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:60419; Asking this server to re-register.
I20240501 19:57:27.275614 15646 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:27.279254 15752 raft_consensus.cc:482] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:27.279373 15752 raft_consensus.cc:504] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.279803 15752 leader_election.cc:288] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.280325 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f46ab4e72478ca29bfca14391db07" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:27.280489 15325 raft_consensus.cc:2454] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.281054 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f46ab4e72478ca29bfca14391db07" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:27.281227 15464 raft_consensus.cc:2454] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:27.281560 15536 leader_election.cc:302] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.283813 15752 raft_consensus.cc:2781] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:27.283895 15752 raft_consensus.cc:482] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:27.283939 15752 raft_consensus.cc:3037] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.285001 15752 raft_consensus.cc:504] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.285429 15752 leader_election.cc:288] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.285946 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f46ab4e72478ca29bfca14391db07" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:27.286098 15325 raft_consensus.cc:3037] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.287112 15325 raft_consensus.cc:2454] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.287778 15464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e3f46ab4e72478ca29bfca14391db07" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:27.287940 15464 raft_consensus.cc:3037] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:27.289078 15533 leader_election.cc:302] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:27.290015 15796 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:27.291591 15464 raft_consensus.cc:2454] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:27.292923 15752 raft_consensus.cc:2781] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:27.293030 15752 raft_consensus.cc:686] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:27.293203 15752 consensus_queue.cc:237] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.299674 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:27.299851 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:27.305814 15716 consensus_queue.cc:1035] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.307459 15510 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:27.309135 15797 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:56505; Asking this server to re-register.
I20240501 19:57:27.309938 15510 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:27.310969 15797 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:27.346931 15752 consensus_queue.cc:1035] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.352169 15723 consensus_queue.cc:1035] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.376498 15056 catalog_manager.cc:3408] renamed table default.table_a94d910a1db245728708bdc7eb0b50b2 [id=4b0264d091414eac90e34e72e5ced992] in HMS: new name default.table_0441f7d080e84fb792251357a57aa031
I20240501 19:57:27.415299 15716 consensus_queue.cc:1035] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.426864 15719 consensus_queue.cc:1035] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.459753 15060 catalog_manager.cc:3408] renamed table default.table_3c7dc746fd214065a974e0e8e2ec6f28 [id=608d4b3e2b584ef8aac50ccdec82ca60] in HMS: new name default.table_8060d3196acf479897cfe5f6f547dba9
I20240501 19:57:27.483008 15723 consensus_queue.cc:1035] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.490506 15723 consensus_queue.cc:1035] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.528591 15059 catalog_manager.cc:3408] renamed table default.table_934e8fcbba9a4c2a9e875db16c2ff080 [id=5ea004652cc64715bc2786eea1248f28] in HMS: new name default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5
I20240501 19:57:27.558012 15752 consensus_queue.cc:1035] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.563630 15752 consensus_queue.cc:1035] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.572263 15723 consensus_queue.cc:1035] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.581220 15752 consensus_queue.cc:1035] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.589792 15058 catalog_manager.cc:3408] renamed table default.table_39fb93a866e54177a673e26dd8325964 [id=06697bc6b1904e478b7551bee08787f8] in HMS: new name default.table_0ef1c033257e4e358ccbe701cce981e2
I20240501 19:57:27.607755 15723 consensus_queue.cc:1035] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.608489 15723 consensus_queue.cc:1035] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.640017 15725 consensus_queue.cc:1035] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.644013 15865 consensus_queue.cc:1035] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.673290 15055 catalog_manager.cc:3408] renamed table default.table_88be21f10fc74ddaab680d288fbb9a13 [id=b299c118e5244019b8ffd6620535d527] in HMS: new name default.table_b0d6857034b54b258f3f6d577cd53e13
I20240501 19:57:27.677831 15868 consensus_queue.cc:1035] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.692449 15725 consensus_queue.cc:1035] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.699090 15701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:27.699277 15701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:27.712671 15057 catalog_manager.cc:2145] created HMS catalog entry for table [id=eb4cd54084b447d89532e8f1f500ba7f]
W20240501 19:57:27.723623 15210 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d695b92bd1a4188b600a0dadd04cc7f consensus state has no leader
I20240501 19:57:27.726011 15752 consensus_queue.cc:1035] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.732692 15723 consensus_queue.cc:1035] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:27.733778 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 539033016933451fa52037d581015148 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.734185 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539033016933451fa52037d581015148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.735191 15057 catalog_manager.cc:5484] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:27.736204 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 539033016933451fa52037d581015148 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.736548 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539033016933451fa52037d581015148. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:57:27.739096 15210 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14b4c5836101423ab760756c51a8c72b consensus state has no leader
I20240501 19:57:27.739571 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 539033016933451fa52037d581015148 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.739859 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 539033016933451fa52037d581015148. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:57:27.744304 15210 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f7c20396c4be498ebb07eb388103afdc consensus state has no leader
I20240501 19:57:27.747380 15910 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:27.747327 15057 catalog_manager.cc:5484] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:27.748529 15910 tablet_bootstrap.cc:654] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.749368 15057 catalog_manager.cc:5484] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } }
I20240501 19:57:27.749792 15057 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:27.750106 15057 catalog_manager.cc:5484] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } }
I20240501 19:57:27.750445 15057 catalog_manager.cc:5484] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:27.750670 15057 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:27.750869 15057 catalog_manager.cc:5484] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } }
I20240501 19:57:27.751086 15057 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]). Expected version 1 got 0
I20240501 19:57:27.756920 15906 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:27.758054 15906 tablet_bootstrap.cc:654] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.760119 15914 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:27.761790 15914 tablet_bootstrap.cc:654] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.763947 15910 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:27.764091 15910 ts_tablet_manager.cc:1367] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:27.764642 15910 raft_consensus.cc:348] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.764792 15910 raft_consensus.cc:374] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.764891 15910 raft_consensus.cc:729] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.765070 15910 consensus_queue.cc:260] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.766520 15906 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:27.766671 15906 ts_tablet_manager.cc:1367] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:27.767264 15906 raft_consensus.cc:348] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.767444 15906 raft_consensus.cc:374] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.767525 15906 raft_consensus.cc:729] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.767720 15906 consensus_queue.cc:260] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.768129 15906 ts_tablet_manager.cc:1398] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:27.768927 15914 tablet_bootstrap.cc:492] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:27.769073 15914 ts_tablet_manager.cc:1367] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:27.769636 15914 raft_consensus.cc:348] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.769820 15914 raft_consensus.cc:374] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.769901 15914 raft_consensus.cc:729] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.770102 15914 consensus_queue.cc:260] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:27.771274 15910 ts_tablet_manager.cc:1398] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:27.773041 15914 ts_tablet_manager.cc:1398] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:27.773936 15918 tablet.cc:1722] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.783149 15918 tablet.cc:1722] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240501 19:57:27.785511 15062 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_ea70d92de3fc40b4b3e8c54d6e5b6899 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:27.786278 15064 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:27.790426 15919 tablet.cc:1722] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.790946 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]). Expected version 1 got 0
I20240501 19:57:27.791066 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:27.791319 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:27.791527 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:27.791630 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ad5586094d51448581f25db4db838305 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:27.793828 15057 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:27.794023 15057 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ad5586094d51448581f25db4db838305 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:27.794127 15057 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:27.794205 15057 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:27.794265 15057 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]). Expected version 1 got 0
I20240501 19:57:27.794981 15918 tablet.cc:1722] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.795504 15919 tablet.cc:1722] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.797124 15057 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:27.798281 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:27.798409 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:27.801334 15925 tablet.cc:1722] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.803326 15918 tablet.cc:1722] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.804040 15057 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8] alter complete (version 1)
I20240501 19:57:27.805323 15928 tablet.cc:1722] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.806481 15920 tablet.cc:1722] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.810046 15920 tablet.cc:1722] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.811625 15057 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992] alter complete (version 1)
I20240501 19:57:27.811949 15920 tablet.cc:1722] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.817512 15926 tablet.cc:1722] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.818497 15918 tablet.cc:1722] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.818611 15057 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28] alter complete (version 1)
I20240501 19:57:27.819209 15929 tablet.cc:1722] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.819298 15925 tablet.cc:1722] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:27.825915 15057 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527] alter complete (version 1)
I20240501 19:57:27.835623 15057 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60] alter complete (version 1)
I20240501 19:57:27.838373 15057 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d695b92bd1a4188b600a0dadd04cc7f from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:27.843820 15057 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 4d695b92bd1a4188b600a0dadd04cc7f
I20240501 19:57:27.844336 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 75b4c1663db04508b03bb074ed9fc788) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:27.844504 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 75b4c1663db04508b03bb074ed9fc788) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:27.844944 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d695b92bd1a4188b600a0dadd04cc7f with delete_type TABLET_DATA_DELETED (replaced by tablet 75b4c1663db04508b03bb074ed9fc788) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:27.845353 15936 tablet_replica.cc:330] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:27.845563 15936 raft_consensus.cc:2229] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:27.845791 15936 raft_consensus.cc:2258] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:27.846495 15936 ts_tablet_manager.cc:1875] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:27.850323 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 75b4c1663db04508b03bb074ed9fc788 (DEFAULT_TABLE table=default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.850659 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b4c1663db04508b03bb074ed9fc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.851778 15936 ts_tablet_manager.cc:1888] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:27.851912 15936 log.cc:1196] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/4d695b92bd1a4188b600a0dadd04cc7f
I20240501 19:57:27.852285 15936 ts_tablet_manager.cc:1909] T 4d695b92bd1a4188b600a0dadd04cc7f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:27.852809 15914 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:27.853056 15935 tablet_replica.cc:330] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:27.853245 15935 raft_consensus.cc:2229] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:27.853283 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 75b4c1663db04508b03bb074ed9fc788 (DEFAULT_TABLE table=default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.853427 15935 raft_consensus.cc:2258] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:27.853542 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b4c1663db04508b03bb074ed9fc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.853814 15914 tablet_bootstrap.cc:654] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.853952 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 75b4c1663db04508b03bb074ed9fc788 (DEFAULT_TABLE table=default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:27.854187 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b4c1663db04508b03bb074ed9fc788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:27.855170 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:27.855403 15057 catalog_manager.cc:6554] ReplaceTablet: tablet 4d695b92bd1a4188b600a0dadd04cc7f deleted and replaced by tablet 75b4c1663db04508b03bb074ed9fc788
I20240501 19:57:27.856051 15914 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:27.856141 15914 ts_tablet_manager.cc:1367] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:27.856207 15906 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:27.856310 15910 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:27.856654 15914 raft_consensus.cc:348] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.856786 15914 raft_consensus.cc:374] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.856820 15914 raft_consensus.cc:729] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.856968 15914 consensus_queue.cc:260] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.857004 15057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_0c593340e8374d2d809dda0943eb4dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:27.857211 15910 tablet_bootstrap.cc:654] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.857211 15906 tablet_bootstrap.cc:654] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:27.857267 15914 ts_tablet_manager.cc:1398] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:27.857280 15934 tablet_replica.cc:330] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:27.857424 15934 raft_consensus.cc:2229] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:27.857789 15934 raft_consensus.cc:2258] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:27.858026 15935 ts_tablet_manager.cc:1875] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:27.858449 15934 ts_tablet_manager.cc:1875] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:27.860164 15935 ts_tablet_manager.cc:1888] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:27.860819 15934 ts_tablet_manager.cc:1888] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:27.860924 15934 log.cc:1196] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/4d695b92bd1a4188b600a0dadd04cc7f
I20240501 19:57:27.861197 15934 ts_tablet_manager.cc:1909] T 4d695b92bd1a4188b600a0dadd04cc7f P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:27.861550 15906 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:27.861653 15906 ts_tablet_manager.cc:1367] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:27.861675 15935 log.cc:1196] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/4d695b92bd1a4188b600a0dadd04cc7f
I20240501 19:57:27.861868 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:27.861943 15935 ts_tablet_manager.cc:1909] T 4d695b92bd1a4188b600a0dadd04cc7f P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:27.862224 15906 raft_consensus.cc:348] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.862347 15906 raft_consensus.cc:374] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.862380 15906 raft_consensus.cc:729] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.862524 15906 consensus_queue.cc:260] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.862596 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 4d695b92bd1a4188b600a0dadd04cc7f (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:27.862809 15906 ts_tablet_manager.cc:1398] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:27.864545 15910 tablet_bootstrap.cc:492] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:27.864650 15910 ts_tablet_manager.cc:1367] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:27.865186 15910 raft_consensus.cc:348] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.865299 15910 raft_consensus.cc:374] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:27.865335 15910 raft_consensus.cc:729] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:27.865475 15910 consensus_queue.cc:260] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:27.865747 15910 ts_tablet_manager.cc:1398] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:28.007988 15865 raft_consensus.cc:482] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.008157 15865 raft_consensus.cc:504] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.008675 15865 leader_election.cc:288] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.009315 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b4c1663db04508b03bb074ed9fc788" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.009467 15598 raft_consensus.cc:2454] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.009829 15261 leader_election.cc:302] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.010012 15865 raft_consensus.cc:2781] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.010054 15865 raft_consensus.cc:482] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.010089 15865 raft_consensus.cc:3037] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.010250 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b4c1663db04508b03bb074ed9fc788" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.010403 15463 raft_consensus.cc:2454] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.011269 15865 raft_consensus.cc:504] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.011682 15865 leader_election.cc:288] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.012166 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b4c1663db04508b03bb074ed9fc788" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.012301 15463 raft_consensus.cc:3037] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.012463 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b4c1663db04508b03bb074ed9fc788" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.012581 15598 raft_consensus.cc:3037] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.013350 15463 raft_consensus.cc:2454] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.013614 15598 raft_consensus.cc:2454] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.014220 15262 leader_election.cc:302] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.014680 15865 raft_consensus.cc:2781] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.014784 15865 raft_consensus.cc:686] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.014940 15865 consensus_queue.cc:237] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.020066 15865 raft_consensus.cc:482] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.020188 15865 raft_consensus.cc:504] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:28.020583 15865 leader_election.cc:288] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.021112 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539033016933451fa52037d581015148" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.021276 15598 raft_consensus.cc:2454] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.021651 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539033016933451fa52037d581015148" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.021764 15261 leader_election.cc:302] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.021792 15463 raft_consensus.cc:2454] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.021916 15865 raft_consensus.cc:2781] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.021970 15865 raft_consensus.cc:482] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.022006 15865 raft_consensus.cc:3037] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.023044 15865 raft_consensus.cc:504] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:28.023424 15865 leader_election.cc:288] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.023938 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539033016933451fa52037d581015148" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.024065 15463 raft_consensus.cc:3037] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.024323 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "539033016933451fa52037d581015148" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.024430 15598 raft_consensus.cc:3037] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.025394 15598 raft_consensus.cc:2454] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.025393 15463 raft_consensus.cc:2454] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.025713 15261 leader_election.cc:302] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.026124 15865 raft_consensus.cc:2781] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.026229 15865 raft_consensus.cc:686] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.026373 15865 consensus_queue.cc:237] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:28.170421 15063 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]
I20240501 19:57:28.173766 15210 catalog_manager.cc:2706] Deleting table default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c] in response to Hive Metastore notification log event 17
I20240501 19:57:28.183112 15210 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e
I20240501 19:57:28.184338 15056 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36756 (request call id 19) took 1481 ms. Trace:
I20240501 19:57:28.184293 15055 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36756 (request call id 12) took 1562 ms. Trace:
I20240501 19:57:28.184427 15056 rpcz_store.cc:276] 0501 19:57:26.702659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.702706 (+ 47us) service_pool.cc:225] Handling call
0501 19:57:26.916048 (+213342us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.183974 (+1267926us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.184130 (+ 156us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:28.184320 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":145023,"Hive Metastore.run_cpu_time_us":863,"Hive Metastore.run_wall_time_us":315072}
I20240501 19:57:28.185701 15056 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
W20240501 19:57:28.186022 15056 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 75b4c1663db04508b03bb074ed9fc788 consensus state has no leader
I20240501 19:57:28.186628 15056 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:28.187382 15056 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:28.188241 15060 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36756 (request call id 11) took 1599 ms. Trace:
I20240501 19:57:28.188298 15060 rpcz_store.cc:276] 0501 19:57:26.588362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.588411 (+ 49us) service_pool.cc:225] Handling call
0501 19:57:26.916282 (+327871us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.187833 (+1271551us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.188029 (+ 196us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:28.188229 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":459971,"Hive Metastore.run_cpu_time_us":602,"Hive Metastore.run_wall_time_us":83303}
I20240501 19:57:28.188469 15060 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d695b92bd1a4188b600a0dadd04cc7f from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.188668 15060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36756 (request call id 28) took 1282 ms. Trace:
I20240501 19:57:28.188709 15060 rpcz_store.cc:276] 0501 19:57:26.906211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.188449 (+1282238us) service_pool.cc:225] Handling call
0501 19:57:28.188657 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:28.188818 15060 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.189020 15060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36756 (request call id 29) took 1282 ms. Trace:
I20240501 19:57:28.189062 15060 rpcz_store.cc:276] 0501 19:57:26.906571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.188802 (+1282231us) service_pool.cc:225] Handling call
0501 19:57:28.189011 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:28.189157 15060 master_service.cc:946] ReplaceTablet: received request to replace tablet 84e74448c8994843bf8b2335669e3713 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.190140 15059 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36756 (request call id 20) took 1430 ms. Trace:
I20240501 19:57:28.190204 15059 rpcz_store.cc:276] 0501 19:57:26.759750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.770490 (+ 10740us) service_pool.cc:225] Handling call
0501 19:57:26.916336 (+145846us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.189804 (+1273468us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.189962 (+ 158us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:28.190128 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":543321,"Hive Metastore.run_cpu_time_us":600,"Hive Metastore.run_wall_time_us":68808}
I20240501 19:57:28.190351 15059 master_service.cc:946] ReplaceTablet: received request to replace tablet ad5586094d51448581f25db4db838305 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.191226 15058 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36756 (request call id 18) took 1530 ms. Trace:
I20240501 19:57:28.191282 15058 rpcz_store.cc:276] 0501 19:57:26.660548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.694052 (+ 33504us) service_pool.cc:225] Handling call
0501 19:57:26.916446 (+222394us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.190879 (+1274433us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.191048 (+ 169us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:28.191212 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":612061,"Hive Metastore.run_cpu_time_us":635,"Hive Metastore.run_wall_time_us":61140}
I20240501 19:57:28.191514 15058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_83b626e1fcb24923aeb39c18f422d4dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.193940 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.195708 15936 tablet_replica.cc:330] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.195853 15936 raft_consensus.cc:2229] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.196213 15936 raft_consensus.cc:2258] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.196761 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.197001 15934 tablet_replica.cc:330] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.197124 15934 raft_consensus.cc:2229] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.197221 15934 raft_consensus.cc:2258] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.197738 15936 ts_tablet_manager.cc:1875] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.197795 15934 ts_tablet_manager.cc:1875] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.199924 15934 ts_tablet_manager.cc:1888] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.199965 15936 ts_tablet_manager.cc:1888] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.200002 15934 log.cc:1196] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/9f2fe4a03e9b4fe9ac80b3c95b5bbc8e
I20240501 19:57:28.200038 15936 log.cc:1196] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/9f2fe4a03e9b4fe9ac80b3c95b5bbc8e
I20240501 19:57:28.200259 15934 ts_tablet_manager.cc:1909] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.200299 15936 ts_tablet_manager.cc:1909] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.200482 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.200663 15935 tablet_replica.cc:330] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.200773 15935 raft_consensus.cc:2229] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.200902 15935 raft_consensus.cc:2258] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.201489 15935 ts_tablet_manager.cc:1875] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.202934 15056 catalog_manager.cc:5484] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.203161 15056 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:28.203516 15935 ts_tablet_manager.cc:1888] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.203580 15935 log.cc:1196] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/9f2fe4a03e9b4fe9ac80b3c95b5bbc8e
I20240501 19:57:28.203823 15935 ts_tablet_manager.cc:1909] T 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.204325 15062 catalog_manager.cc:2145] created HMS catalog entry for table [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:28.205360 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (table default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]) successfully deleted
I20240501 19:57:28.205852 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (table default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]) successfully deleted
I20240501 19:57:28.184480 15055 rpcz_store.cc:276] 0501 19:57:26.621637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.621684 (+ 47us) service_pool.cc:225] Handling call
0501 19:57:26.916465 (+294781us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.183733 (+1267268us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.184046 (+ 313us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:28.184274 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":673192,"Hive Metastore.run_cpu_time_us":474,"Hive Metastore.run_wall_time_us":83443}
I20240501 19:57:28.207149 15055 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0ca797a676d54ea89cfea3963a75b1d0" } modify_external_catalogs: true
I20240501 19:57:28.207307 15055 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0ca797a676d54ea89cfea3963a75b1d0" } modify_external_catalogs: true
I20240501 19:57:28.210673 15059 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet ad5586094d51448581f25db4db838305
I20240501 19:57:28.210970 15059 catalog_manager.cc:6554] ReplaceTablet: tablet ad5586094d51448581f25db4db838305 deleted and replaced by tablet dce88e9c62204ac8a67be46336f09865
I20240501 19:57:28.211028 15059 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36756 (request call id 31) took 1303 ms. Trace:
I20240501 19:57:28.211066 15059 rpcz_store.cc:276] 0501 19:57:26.907637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.190335 (+1282698us) service_pool.cc:225] Handling call
0501 19:57:28.211019 (+ 20684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.193068 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.193135 (+ 67us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.193139 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:28.193141 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:28.193209 (+ 68us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.193216 (+ 7us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.193219 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.193258 (+ 39us) tablet.cc:699] Row locks acquired
0501 19:57:28.193259 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:28.193281 (+ 22us) write_op.cc:270] Start()
0501 19:57:28.193299 (+ 18us) write_op.cc:276] Timestamp: P: 1714593448193251 usec, L: 0
0501 19:57:28.193301 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.193398 (+ 97us) log.cc:842] Serialized 655 byte log entry
0501 19:57:28.209700 (+ 16302us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.210236 (+ 536us) write_op.cc:301] APPLY: starting
0501 19:57:28.210245 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.210267 (+ 22us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.210280 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.210336 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.210366 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 19:57:28.210370 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.210412 (+ 42us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.210444 (+ 32us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.210452 (+ 8us) write_op.cc:446] Released schema lock
0501 19:57:28.210456 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.210508 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.210510 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":514,"apply.run_cpu_time_us":306,"apply.run_wall_time_us":306,"num_ops":2,"prepare.queue_time_us":2209,"prepare.run_cpu_time_us":518,"prepare.run_wall_time_us":521,"raft.queue_time_us":4297,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":16390,"thread_start_us":105,"threads_started":1}]]}
I20240501 19:57:28.211385 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5586094d51448581f25db4db838305 with delete_type TABLET_DATA_DELETED (replaced by tablet dce88e9c62204ac8a67be46336f09865) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.211576 15059 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_b7b25ac22dba40c9aa0a509437741d50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.212342 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5586094d51448581f25db4db838305 with delete_type TABLET_DATA_DELETED (replaced by tablet dce88e9c62204ac8a67be46336f09865) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.212927 15934 tablet_replica.cc:330] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.214205 15056 catalog_manager.cc:5484] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:28.214656 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5586094d51448581f25db4db838305 with delete_type TABLET_DATA_DELETED (replaced by tablet dce88e9c62204ac8a67be46336f09865) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.214879 15935 tablet_replica.cc:330] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.215003 15935 raft_consensus.cc:2229] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.215116 15935 raft_consensus.cc:2258] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.215751 15935 ts_tablet_manager.cc:1875] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.215871 15934 raft_consensus.cc:2229] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.216042 15934 raft_consensus.cc:2258] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.216616 15934 ts_tablet_manager.cc:1875] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.218811 15935 ts_tablet_manager.cc:1888] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.220916 15934 ts_tablet_manager.cc:1888] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.221066 15935 log.cc:1196] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/ad5586094d51448581f25db4db838305
I20240501 19:57:28.221388 15935 ts_tablet_manager.cc:1909] T ad5586094d51448581f25db4db838305 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.221485 15934 log.cc:1196] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/ad5586094d51448581f25db4db838305
I20240501 19:57:28.221778 15934 ts_tablet_manager.cc:1909] T ad5586094d51448581f25db4db838305 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.224380 15936 tablet_replica.cc:330] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.224522 15936 raft_consensus.cc:2229] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.224715 15936 raft_consensus.cc:2258] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.225338 15936 ts_tablet_manager.cc:1875] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.227696 15936 ts_tablet_manager.cc:1888] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.227803 15936 log.cc:1196] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/ad5586094d51448581f25db4db838305
I20240501 19:57:28.228098 15936 ts_tablet_manager.cc:1909] T ad5586094d51448581f25db4db838305 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.216709 15062 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 23) took 1393 ms. Trace:
I20240501 19:57:28.228917 15062 rpcz_store.cc:276] 0501 19:57:26.823006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:27.024251 (+201245us) service_pool.cc:225] Handling call
0501 19:57:27.785618 (+761367us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:27.785969 (+ 351us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.204318 (+418349us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.216532 (+ 12214us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.216624 (+ 92us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.216694 (+ 70us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.208452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.208546 (+ 94us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.208551 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:28.208552 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:28.208626 (+ 74us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.208635 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.208638 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.208693 (+ 55us) tablet.cc:699] Row locks acquired
0501 19:57:28.208695 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:28.208724 (+ 29us) write_op.cc:270] Start()
0501 19:57:28.208760 (+ 36us) write_op.cc:276] Timestamp: P: 1714593448208691 usec, L: 0
0501 19:57:28.208762 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.208929 (+ 167us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.216014 (+ 7085us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.216094 (+ 80us) write_op.cc:301] APPLY: starting
0501 19:57:28.216108 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.216134 (+ 26us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.216149 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.216237 (+ 88us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.216267 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.216272 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:28.216331 (+ 59us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.216381 (+ 50us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.216394 (+ 13us) write_op.cc:446] Released schema lock
0501 19:57:28.216399 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.216445 (+ 46us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.216450 (+ 5us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":387340,"Hive Metastore.run_cpu_time_us":206,"Hive Metastore.run_wall_time_us":30757,"child_traces":[["op",{"apply.queue_time_us":45,"apply.run_cpu_time_us":397,"apply.run_wall_time_us":396,"num_ops":2,"prepare.queue_time_us":3732,"prepare.run_cpu_time_us":536,"prepare.run_wall_time_us":537,"replication_time_us":7232}]]}
I20240501 19:57:28.230403 15062 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0c03b0e181d140c88754b558e4fb4393" } modify_external_catalogs: true
I20240501 19:57:28.230542 15062 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0c03b0e181d140c88754b558e4fb4393" } modify_external_catalogs: true
I20240501 19:57:28.217327 15060 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 84e74448c8994843bf8b2335669e3713
I20240501 19:57:28.231285 15060 catalog_manager.cc:6554] ReplaceTablet: tablet 84e74448c8994843bf8b2335669e3713 deleted and replaced by tablet ccdc0cb3c51e4f87bc9f1d1a03984591
I20240501 19:57:28.231361 15060 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36756 (request call id 30) took 1324 ms. Trace:
I20240501 19:57:28.224169 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 9f2fe4a03e9b4fe9ac80b3c95b5bbc8e (table default.table_40db4cea1aad411796b548b57c595842 [id=0ef7499159164fcf9c5b0d5e4484153c]) successfully deleted
I20240501 19:57:28.232072 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet ad5586094d51448581f25db4db838305 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:28.224196 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet ad5586094d51448581f25db4db838305 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:28.224651 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet ad5586094d51448581f25db4db838305 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:28.232761 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 84e74448c8994843bf8b2335669e3713 with delete_type TABLET_DATA_DELETED (replaced by tablet ccdc0cb3c51e4f87bc9f1d1a03984591) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.232977 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 84e74448c8994843bf8b2335669e3713 with delete_type TABLET_DATA_DELETED (replaced by tablet ccdc0cb3c51e4f87bc9f1d1a03984591) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.233171 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 84e74448c8994843bf8b2335669e3713 with delete_type TABLET_DATA_DELETED (replaced by tablet ccdc0cb3c51e4f87bc9f1d1a03984591) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.233393 15935 tablet_replica.cc:330] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.231402 15060 rpcz_store.cc:276] 0501 19:57:26.906680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.189142 (+1282462us) service_pool.cc:225] Handling call
0501 19:57:28.231351 (+ 42209us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.192482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.192559 (+ 77us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.192564 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:28.192565 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:28.192646 (+ 81us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.192656 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.192659 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.192713 (+ 54us) tablet.cc:699] Row locks acquired
0501 19:57:28.192715 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:28.192744 (+ 29us) write_op.cc:270] Start()
0501 19:57:28.192771 (+ 27us) write_op.cc:276] Timestamp: P: 1714593448192712 usec, L: 0
0501 19:57:28.192773 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.192949 (+ 176us) log.cc:842] Serialized 655 byte log entry
0501 19:57:28.209650 (+ 16701us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.209768 (+ 118us) write_op.cc:301] APPLY: starting
0501 19:57:28.209783 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.209818 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.209833 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.209914 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.209956 (+ 42us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0501 19:57:28.209960 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.210019 (+ 59us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.210083 (+ 64us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.210099 (+ 16us) write_op.cc:446] Released schema lock
0501 19:57:28.210104 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.210155 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.210157 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":443,"apply.run_wall_time_us":442,"num_ops":2,"prepare.queue_time_us":2701,"prepare.run_cpu_time_us":560,"prepare.run_wall_time_us":560,"raft.queue_time_us":4549,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":385,"replication_time_us":16858}]]}
I20240501 19:57:28.233616 15934 tablet_replica.cc:330] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.234090 15935 raft_consensus.cc:2229] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.234364 15060 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_31c7944e4b41499683c266b366d0711f" } modify_external_catalogs: true
I20240501 19:57:28.234493 15934 raft_consensus.cc:2229] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.234544 15060 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_31c7944e4b41499683c266b366d0711f" } modify_external_catalogs: true
I20240501 19:57:28.234663 15934 raft_consensus.cc:2258] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.234900 15935 raft_consensus.cc:2258] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.235554 15934 ts_tablet_manager.cc:1875] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.236438 15935 ts_tablet_manager.cc:1875] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.242453 15934 ts_tablet_manager.cc:1888] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.242545 15934 log.cc:1196] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/84e74448c8994843bf8b2335669e3713
I20240501 19:57:28.244097 15064 catalog_manager.cc:2145] created HMS catalog entry for table [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:28.244755 15935 ts_tablet_manager.cc:1888] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.244874 15935 log.cc:1196] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/84e74448c8994843bf8b2335669e3713
I20240501 19:57:28.245170 15935 ts_tablet_manager.cc:1909] T 84e74448c8994843bf8b2335669e3713 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.245878 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.247076 15934 ts_tablet_manager.cc:1909] T 84e74448c8994843bf8b2335669e3713 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.250630 15064 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 24) took 1389 ms. Trace:
I20240501 19:57:28.250712 15064 rpcz_store.cc:276] 0501 19:57:26.860817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:27.026233 (+165416us) service_pool.cc:225] Handling call
0501 19:57:27.786356 (+760123us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:27.786653 (+ 297us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.244088 (+457435us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.250469 (+ 6381us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.250553 (+ 84us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.250614 (+ 61us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.244599 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.244696 (+ 97us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.244699 (+ 3us) write_op.cc:427] Acquired schema lock
0501 19:57:28.244700 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:28.244765 (+ 65us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.244773 (+ 8us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.244776 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.244830 (+ 54us) tablet.cc:699] Row locks acquired
0501 19:57:28.244831 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:28.245018 (+ 187us) write_op.cc:270] Start()
0501 19:57:28.245043 (+ 25us) write_op.cc:276] Timestamp: P: 1714593448244978 usec, L: 0
0501 19:57:28.245044 (+ 1us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.245166 (+ 122us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.247974 (+ 2808us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.248073 (+ 99us) write_op.cc:301] APPLY: starting
0501 19:57:28.248084 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.248112 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.248125 (+ 13us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.248200 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.248228 (+ 28us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.248233 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:28.248278 (+ 45us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.248308 (+ 30us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.248316 (+ 8us) write_op.cc:446] Released schema lock
0501 19:57:28.248320 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.248360 (+ 40us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.248362 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":417478,"Hive Metastore.run_cpu_time_us":217,"Hive Metastore.run_wall_time_us":39853,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":320,"apply.run_wall_time_us":319,"num_ops":2,"prepare.queue_time_us":90,"prepare.run_cpu_time_us":542,"prepare.run_wall_time_us":661,"raft.queue_time_us":1071,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":260,"replication_time_us":2916}]]}
I20240501 19:57:28.251157 15064 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.251531 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.251859 15306 tablet_service.cc:1467] Processing CreateTablet for tablet dce88e9c62204ac8a67be46336f09865 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.252125 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce88e9c62204ac8a67be46336f09865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.252233 15444 tablet_service.cc:1467] Processing CreateTablet for tablet dce88e9c62204ac8a67be46336f09865 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.252521 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce88e9c62204ac8a67be46336f09865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.254328 15914 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.254810 15906 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.255300 15914 tablet_bootstrap.cc:654] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.255700 15906 tablet_bootstrap.cc:654] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.260146 15579 tablet_service.cc:1467] Processing CreateTablet for tablet dce88e9c62204ac8a67be46336f09865 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.260491 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce88e9c62204ac8a67be46336f09865. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.261029 15598 raft_consensus.cc:1261] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.261572 15868 consensus_queue.cc:1035] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.262739 15910 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.263597 15910 tablet_bootstrap.cc:654] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.265543 15906 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.265631 15463 raft_consensus.cc:1261] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.265653 15906 ts_tablet_manager.cc:1367] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:28.266147 15906 raft_consensus.cc:348] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.266258 15906 raft_consensus.cc:374] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.266290 15906 raft_consensus.cc:729] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.266407 15906 consensus_queue.cc:260] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.266691 15906 ts_tablet_manager.cc:1398] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.269595 15063 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36756 (request call id 22) took 1451 ms. Trace:
I20240501 19:57:28.269681 15063 rpcz_store.cc:276] 0501 19:57:26.817903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:26.915213 (+ 97310us) service_pool.cc:225] Handling call
0501 19:57:27.071456 (+156243us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.269573 (+1198117us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":645179,"Hive Metastore.run_cpu_time_us":187,"Hive Metastore.run_wall_time_us":453601}
W20240501 19:57:28.269800 15057 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0c593340e8374d2d809dda0943eb4dd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.270027 15063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_758f18364c874f0a824123020aad0241"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.270349 15868 consensus_queue.cc:1035] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.271000 15910 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.271086 15910 ts_tablet_manager.cc:1367] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:28.271603 15910 raft_consensus.cc:348] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.271704 15910 raft_consensus.cc:374] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.271731 15910 raft_consensus.cc:729] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.271848 15910 consensus_queue.cc:260] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.272094 15910 ts_tablet_manager.cc:1398] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.277976 15920 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.278331 15914 tablet_bootstrap.cc:492] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.278422 15914 ts_tablet_manager.cc:1367] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240501 19:57:28.278759 15877 mvcc.cc:205] Tried to move back new op lower bound from 7022974764072222720 to 7022974763072520192. Current Snapshot: MvccSnapshot[applied={T|T < 7022974764072222720}]
I20240501 19:57:28.278896 15914 raft_consensus.cc:348] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.278990 15914 raft_consensus.cc:374] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.279017 15914 raft_consensus.cc:729] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.279126 15914 consensus_queue.cc:260] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.279361 15914 ts_tablet_manager.cc:1398] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.282923 15932 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.286334 15927 mvcc.cc:205] Tried to move back new op lower bound from 7022974764072222720 to 7022974763072520192. Current Snapshot: MvccSnapshot[applied={T|T < 7022974764072222720}]
I20240501 19:57:28.286506 15918 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.294512 15579 tablet_service.cc:1467] Processing CreateTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.294714 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (DEFAULT_TABLE table=default.table_ea70d92de3fc40b4b3e8c54d6e5b6899 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.294855 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdc0cb3c51e4f87bc9f1d1a03984591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.295001 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6cb71ea54e4cc2985b9d607109aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.296365 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (DEFAULT_TABLE table=default.table_ea70d92de3fc40b4b3e8c54d6e5b6899 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.296636 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6cb71ea54e4cc2985b9d607109aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.297245 15910 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.298300 15910 tablet_bootstrap.cc:654] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.298974 15914 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.299957 15914 tablet_bootstrap.cc:654] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.302299 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (DEFAULT_TABLE table=default.table_ea70d92de3fc40b4b3e8c54d6e5b6899 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.302636 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6cb71ea54e4cc2985b9d607109aba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.302938 15910 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.303031 15910 ts_tablet_manager.cc:1367] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:28.303534 15910 raft_consensus.cc:348] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.303643 15910 raft_consensus.cc:374] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.303671 15910 raft_consensus.cc:729] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.303799 15910 consensus_queue.cc:260] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.304054 15910 ts_tablet_manager.cc:1398] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.304236 15910 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.305156 15910 tablet_bootstrap.cc:654] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.306255 15910 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.306324 15910 ts_tablet_manager.cc:1367] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:28.306733 15910 raft_consensus.cc:348] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.306828 15910 raft_consensus.cc:374] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.306903 15910 raft_consensus.cc:729] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.307039 15910 consensus_queue.cc:260] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.307296 15910 ts_tablet_manager.cc:1398] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.308960 15906 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.309128 15306 tablet_service.cc:1467] Processing CreateTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.309410 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdc0cb3c51e4f87bc9f1d1a03984591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.309895 15906 tablet_bootstrap.cc:654] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.311911 15444 tablet_service.cc:1467] Processing CreateTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.312161 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccdc0cb3c51e4f87bc9f1d1a03984591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.313592 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0ca8638f14b89a2041c10a2e2206d (DEFAULT_TABLE table=default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.313834 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0ca8638f14b89a2041c10a2e2206d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.315306 15056 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.315410 15056 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
W20240501 19:57:28.315791 15056 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet dce88e9c62204ac8a67be46336f09865 consensus state has no leader
I20240501 19:57:28.316640 15056 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992]). Expected version 1 got 0
I20240501 19:57:28.316727 15056 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.317348 15056 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_8d1020ff0b064f7199ad44d673369f79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.318035 15910 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.319023 15910 tablet_bootstrap.cc:654] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.319690 15061 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]
I20240501 19:57:28.321823 15868 raft_consensus.cc:482] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.321919 15868 raft_consensus.cc:504] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.322299 15868 leader_election.cc:288] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.322322 15936 tablet_replica.cc:330] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.322499 15936 raft_consensus.cc:2229] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.322636 15936 raft_consensus.cc:2258] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.322724 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce88e9c62204ac8a67be46336f09865" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.322796 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce88e9c62204ac8a67be46336f09865" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.322844 15463 raft_consensus.cc:2454] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.322978 15598 raft_consensus.cc:2454] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.323128 15262 leader_election.cc:302] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.323269 15936 ts_tablet_manager.cc:1875] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.323290 15868 raft_consensus.cc:2781] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.323338 15868 raft_consensus.cc:482] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.323372 15868 raft_consensus.cc:3037] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.324429 15868 raft_consensus.cc:504] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.324769 15868 leader_election.cc:288] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.325273 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce88e9c62204ac8a67be46336f09865" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.325387 15598 raft_consensus.cc:3037] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.325755 15210 catalog_manager.cc:2706] Deleting table default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec] in response to Hive Metastore notification log event 20
I20240501 19:57:28.326881 15598 raft_consensus.cc:2454] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.327176 15261 leader_election.cc:302] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.327353 15868 raft_consensus.cc:2781] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.327420 15868 raft_consensus.cc:686] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.327569 15868 consensus_queue.cc:237] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.330085 15910 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.330178 15910 ts_tablet_manager.cc:1367] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:28.330642 15910 raft_consensus.cc:348] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.330747 15910 raft_consensus.cc:374] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.330780 15910 raft_consensus.cc:729] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.330929 15910 consensus_queue.cc:260] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.332423 15910 ts_tablet_manager.cc:1398] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:28.333750 15906 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.333843 15906 ts_tablet_manager.cc:1367] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20240501 19:57:28.334511 15906 raft_consensus.cc:348] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.334619 15906 raft_consensus.cc:374] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.334648 15906 raft_consensus.cc:729] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.334764 15906 consensus_queue.cc:260] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.335017 15906 ts_tablet_manager.cc:1398] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.335197 15906 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.335322 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce88e9c62204ac8a67be46336f09865" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.335431 15463 raft_consensus.cc:3037] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.336337 15906 tablet_bootstrap.cc:654] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.336521 15463 raft_consensus.cc:2454] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.336604 15936 ts_tablet_manager.cc:1888] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.336660 15936 log.cc:1196] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/84e74448c8994843bf8b2335669e3713
I20240501 19:57:28.336978 15936 ts_tablet_manager.cc:1909] T 84e74448c8994843bf8b2335669e3713 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.338346 15210 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34
I20240501 19:57:28.338795 15914 tablet_bootstrap.cc:492] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.338896 15914 ts_tablet_manager.cc:1367] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.000s
W20240501 19:57:28.339139 15059 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b7b25ac22dba40c9aa0a509437741d50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.339403 15914 raft_consensus.cc:348] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.339524 15914 raft_consensus.cc:374] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.339556 15914 raft_consensus.cc:729] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.339685 15914 consensus_queue.cc:260] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.339977 15914 ts_tablet_manager.cc:1398] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.340178 15914 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.341049 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.341231 15935 tablet_replica.cc:330] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.341359 15935 raft_consensus.cc:2229] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.341572 15935 raft_consensus.cc:2258] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.341807 15914 tablet_bootstrap.cc:654] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.342276 15935 ts_tablet_manager.cc:1875] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 19:57:28.342739 15064 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.344580 15935 ts_tablet_manager.cc:1888] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.344650 15935 log.cc:1196] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/9e47dc8f2e6b4e4e9b8d8a7b6f140e34
W20240501 19:57:28.345227 15058 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_83b626e1fcb24923aeb39c18f422d4dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:28.346617 15050 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dce88e9c62204ac8a67be46336f09865 consensus state has no leader
I20240501 19:57:28.350421 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.350643 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.350710 15935 ts_tablet_manager.cc:1909] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.350829 15934 tablet_replica.cc:330] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.350975 15934 raft_consensus.cc:2229] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.351114 15934 raft_consensus.cc:2258] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.351511 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (table default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]) successfully deleted
I20240501 19:57:28.351778 15934 ts_tablet_manager.cc:1875] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.352982 15906 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.353076 15906 ts_tablet_manager.cc:1367] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 19:57:28.353578 15906 raft_consensus.cc:348] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.353684 15906 raft_consensus.cc:374] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.353714 15906 raft_consensus.cc:729] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.353853 15906 consensus_queue.cc:260] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.354122 15906 ts_tablet_manager.cc:1398] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.354112 15934 ts_tablet_manager.cc:1888] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.354398 15934 log.cc:1196] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/9e47dc8f2e6b4e4e9b8d8a7b6f140e34
I20240501 19:57:28.354678 15934 ts_tablet_manager.cc:1909] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.355508 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (table default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]) successfully deleted
I20240501 19:57:28.358208 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0ca8638f14b89a2041c10a2e2206d (DEFAULT_TABLE table=default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.358516 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0ca8638f14b89a2041c10a2e2206d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.361088 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 71c0ca8638f14b89a2041c10a2e2206d (DEFAULT_TABLE table=default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.361431 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71c0ca8638f14b89a2041c10a2e2206d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.365010 15936 tablet_replica.cc:330] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.365165 15936 raft_consensus.cc:2229] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.365160 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 84e74448c8994843bf8b2335669e3713 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.365294 15936 raft_consensus.cc:2258] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.365873 15936 ts_tablet_manager.cc:1875] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.368136 15936 ts_tablet_manager.cc:1888] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.368220 15936 log.cc:1196] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/9e47dc8f2e6b4e4e9b8d8a7b6f140e34
I20240501 19:57:28.368489 15936 ts_tablet_manager.cc:1909] T 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.369232 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 9e47dc8f2e6b4e4e9b8d8a7b6f140e34 (table default.table_b78cbc9abde043b6a52288415772661c [id=a8f768afdece4f33a1089d673dc188ec]) successfully deleted
I20240501 19:57:28.372759 15906 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.374076 15906 tablet_bootstrap.cc:654] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.374460 15914 tablet_bootstrap.cc:492] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.374552 15914 ts_tablet_manager.cc:1367] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.000s
I20240501 19:57:28.375032 15914 raft_consensus.cc:348] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.375160 15914 raft_consensus.cc:374] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.375212 15914 raft_consensus.cc:729] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.375375 15914 consensus_queue.cc:260] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.375995 15057 catalog_manager.cc:2145] created HMS catalog entry for table [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:28.378901 15914 ts_tablet_manager.cc:1398] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:28.381155 15057 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 27) took 1477 ms. Trace:
I20240501 19:57:28.381234 15057 rpcz_store.cc:276] 0501 19:57:26.903974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:27.856890 (+952916us) service_pool.cc:225] Handling call
0501 19:57:28.269883 (+412993us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.270215 (+ 332us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.375987 (+105772us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.380937 (+ 4950us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.381035 (+ 98us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.381135 (+ 100us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.376477 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.376572 (+ 95us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.376577 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:28.376578 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:28.376644 (+ 66us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.376653 (+ 9us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.376656 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.376711 (+ 55us) tablet.cc:699] Row locks acquired
0501 19:57:28.376712 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:28.376740 (+ 28us) write_op.cc:270] Start()
0501 19:57:28.376775 (+ 35us) write_op.cc:276] Timestamp: P: 1714593448376693 usec, L: 0
0501 19:57:28.376777 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.376982 (+ 205us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.379974 (+ 2992us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.380064 (+ 90us) write_op.cc:301] APPLY: starting
0501 19:57:28.380079 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.380115 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.380132 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.380208 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.380239 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.380244 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:28.380305 (+ 61us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.380353 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.380368 (+ 15us) write_op.cc:446] Released schema lock
0501 19:57:28.380373 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.380429 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.380431 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":54964,"Hive Metastore.run_cpu_time_us":216,"Hive Metastore.run_wall_time_us":50572,"child_traces":[["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":407,"apply.run_wall_time_us":407,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":588,"prepare.run_wall_time_us":588,"raft.queue_time_us":838,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":281,"replication_time_us":3177}]]}
I20240501 19:57:28.381557 15906 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.381656 15906 ts_tablet_manager.cc:1367] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:28.381908 15914 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.382193 15906 raft_consensus.cc:348] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.382340 15906 raft_consensus.cc:374] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.382372 15906 raft_consensus.cc:729] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.383037 15914 tablet_bootstrap.cc:654] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.383085 15906 consensus_queue.cc:260] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.383253 15057 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.383430 15906 ts_tablet_manager.cc:1398] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:57:28.384369 15914 tablet_bootstrap.cc:492] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.384435 15914 ts_tablet_manager.cc:1367] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:28.384774 15914 raft_consensus.cc:348] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.384912 15914 raft_consensus.cc:374] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.384958 15914 raft_consensus.cc:729] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.385123 15914 consensus_queue.cc:260] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.388962 15914 ts_tablet_manager.cc:1398] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
W20240501 19:57:28.395984 15056 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_8d1020ff0b064f7199ad44d673369f79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:28.396351 15063 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_758f18364c874f0a824123020aad0241 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.396929 15061 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36756 (request call id 25) took 1512 ms. Trace:
I20240501 19:57:28.396992 15061 rpcz_store.cc:276] 0501 19:57:26.883989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:27.089692 (+205703us) service_pool.cc:225] Handling call
0501 19:57:28.183840 (+1094148us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.396912 (+213072us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":62949,"Hive Metastore.run_cpu_time_us":137,"Hive Metastore.run_wall_time_us":72557}
I20240501 19:57:28.397939 15061 catalog_manager.cc:5484] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.398169 15061 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.398344 15061 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.398447 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 0f838413cd7a4975918e88551c465f18 (DEFAULT_TABLE table=default.table_0c593340e8374d2d809dda0943eb4dd7 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.398759 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f838413cd7a4975918e88551c465f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.401559 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 0f838413cd7a4975918e88551c465f18 (DEFAULT_TABLE table=default.table_0c593340e8374d2d809dda0943eb4dd7 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.401865 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f838413cd7a4975918e88551c465f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.402383 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 0f838413cd7a4975918e88551c465f18 (DEFAULT_TABLE table=default.table_0c593340e8374d2d809dda0943eb4dd7 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.402746 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f838413cd7a4975918e88551c465f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.406301 15910 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.407356 15910 tablet_bootstrap.cc:654] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.407931 15914 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.409291 15914 tablet_bootstrap.cc:654] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:28.410142 15061 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ccdc0cb3c51e4f87bc9f1d1a03984591 consensus state has no leader
I20240501 19:57:28.410989 15061 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.411615 15061 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_090e62cd0c6c43eb9c44ada659e01c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.417560 15910 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.417670 15910 ts_tablet_manager.cc:1367] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:28.418190 15910 raft_consensus.cc:348] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.418313 15910 raft_consensus.cc:374] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.418347 15910 raft_consensus.cc:729] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.418473 15910 consensus_queue.cc:260] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.418751 15910 ts_tablet_manager.cc:1398] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.419118 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.419260 15057 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.420759 15057 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_d8ca1b5399e14ad98016d79abbc191df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.424338 15906 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.425475 15906 tablet_bootstrap.cc:654] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.427364 15914 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.427557 15914 ts_tablet_manager.cc:1367] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:57:28.427807 15463 raft_consensus.cc:1261] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.428253 15865 consensus_queue.cc:1035] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.428261 15914 raft_consensus.cc:348] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.428618 15914 raft_consensus.cc:374] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.428697 15914 raft_consensus.cc:729] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.428926 15914 consensus_queue.cc:260] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.431216 15914 ts_tablet_manager.cc:1398] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:28.435390 15598 raft_consensus.cc:1261] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.436039 15865 consensus_queue.cc:1035] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.442239 15920 tablet.cc:1722] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.444532 15059 catalog_manager.cc:2145] created HMS catalog entry for table [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:28.444815 15932 tablet.cc:1722] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.446215 15921 mvcc.cc:205] Tried to move back new op lower bound from 7022974764756299776 to 7022974764351361024. Current Snapshot: MvccSnapshot[applied={T|T < 7022974764756299776}]
I20240501 19:57:28.449985 15059 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 34) took 1431 ms. Trace:
I20240501 19:57:28.450065 15059 rpcz_store.cc:276] 0501 19:57:27.018183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.211456 (+1193273us) service_pool.cc:225] Handling call
0501 19:57:28.339220 (+127764us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.339565 (+ 345us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.444522 (+104957us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.449792 (+ 5270us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.449890 (+ 98us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.449968 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.445111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.445211 (+ 100us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.445216 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:28.445217 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:28.445283 (+ 66us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.445293 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.445297 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.445353 (+ 56us) tablet.cc:699] Row locks acquired
0501 19:57:28.445355 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:28.445380 (+ 25us) write_op.cc:270] Start()
0501 19:57:28.445402 (+ 22us) write_op.cc:276] Timestamp: P: 1714593448445328 usec, L: 0
0501 19:57:28.445404 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.445541 (+ 137us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.449126 (+ 3585us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.449214 (+ 88us) write_op.cc:301] APPLY: starting
0501 19:57:28.449231 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.449263 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.449278 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.449352 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.449384 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.449390 (+ 6us) write_op.cc:312] APPLY: finished
0501 19:57:28.449458 (+ 68us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.449510 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.449523 (+ 13us) write_op.cc:446] Released schema lock
0501 19:57:28.449528 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.449580 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.449582 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":46943,"Hive Metastore.run_cpu_time_us":201,"Hive Metastore.run_wall_time_us":57713,"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":403,"apply.run_wall_time_us":402,"num_ops":2,"prepare.queue_time_us":94,"prepare.run_cpu_time_us":513,"prepare.run_wall_time_us":512,"raft.queue_time_us":881,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":338,"replication_time_us":3703}]]}
I20240501 19:57:28.451855 15059 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:28.459020 15906 tablet_bootstrap.cc:492] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.459136 15906 ts_tablet_manager.cc:1367] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240501 19:57:28.459630 15906 raft_consensus.cc:348] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.459755 15906 raft_consensus.cc:374] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.459789 15906 raft_consensus.cc:729] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.459918 15906 consensus_queue.cc:260] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.460242 15906 ts_tablet_manager.cc:1398] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:28.462447 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (DEFAULT_TABLE table=default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.462775 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed5f03a8ff34913a59c475bd7ec83cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.463101 15918 tablet.cc:1722] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.465018 15910 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.466066 15910 tablet_bootstrap.cc:654] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.468582 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (DEFAULT_TABLE table=default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.468859 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed5f03a8ff34913a59c475bd7ec83cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.469377 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (DEFAULT_TABLE table=default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.470010 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed5f03a8ff34913a59c475bd7ec83cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.472151 15906 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.473712 15906 tablet_bootstrap.cc:654] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.475129 15906 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.475225 15906 ts_tablet_manager.cc:1367] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:28.475703 15914 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.475701 15906 raft_consensus.cc:348] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.475808 15906 raft_consensus.cc:374] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.475838 15906 raft_consensus.cc:729] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.475967 15906 consensus_queue.cc:260] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.476248 15906 ts_tablet_manager.cc:1398] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:28.476680 15914 tablet_bootstrap.cc:654] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.478735 15910 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.478843 15910 ts_tablet_manager.cc:1367] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:28.478883 15914 tablet_bootstrap.cc:492] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.478976 15914 ts_tablet_manager.cc:1367] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:28.479346 15910 raft_consensus.cc:348] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.479398 15914 raft_consensus.cc:348] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.479460 15910 raft_consensus.cc:374] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.479487 15914 raft_consensus.cc:374] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.479494 15910 raft_consensus.cc:729] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.479509 15914 raft_consensus.cc:729] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.479620 15914 consensus_queue.cc:260] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.479660 15910 consensus_queue.cc:260] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.479882 15914 ts_tablet_manager.cc:1398] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:28.483273 15910 ts_tablet_manager.cc:1398] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:28.490979 15059 master_service.cc:946] ReplaceTablet: received request to replace tablet f7c20396c4be498ebb07eb388103afdc from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.492905 15865 raft_consensus.cc:482] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.493029 15865 raft_consensus.cc:504] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.493482 15865 leader_election.cc:288] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.494258 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f838413cd7a4975918e88551c465f18" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.494410 15463 raft_consensus.cc:2454] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.494459 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f838413cd7a4975918e88551c465f18" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.494588 15598 raft_consensus.cc:2454] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.494889 15262 leader_election.cc:302] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.495044 15868 raft_consensus.cc:482] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.495117 15868 raft_consensus.cc:504] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.495323 15865 raft_consensus.cc:2781] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.495370 15865 raft_consensus.cc:482] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.495409 15865 raft_consensus.cc:3037] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.495496 15868 leader_election.cc:288] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.495995 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdc0cb3c51e4f87bc9f1d1a03984591" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.496109 15463 raft_consensus.cc:2454] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.496454 15262 leader_election.cc:302] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.496553 15865 raft_consensus.cc:504] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.496692 15868 raft_consensus.cc:2781] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.496738 15868 raft_consensus.cc:482] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.496776 15868 raft_consensus.cc:3037] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.496912 15865 leader_election.cc:288] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.497427 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdc0cb3c51e4f87bc9f1d1a03984591" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.497565 15598 raft_consensus.cc:2454] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.497846 15868 raft_consensus.cc:504] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.498160 15868 leader_election.cc:288] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.498198 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f838413cd7a4975918e88551c465f18" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.498301 15463 raft_consensus.cc:3037] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.498369 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f838413cd7a4975918e88551c465f18" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.498485 15599 raft_consensus.cc:3037] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.499267 15463 raft_consensus.cc:2454] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.499472 15599 raft_consensus.cc:2454] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.499888 15261 leader_election.cc:302] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.499908 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdc0cb3c51e4f87bc9f1d1a03984591" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.500013 15599 raft_consensus.cc:3037] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.500414 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccdc0cb3c51e4f87bc9f1d1a03984591" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.500520 15463 raft_consensus.cc:3037] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.500806 15059 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet f7c20396c4be498ebb07eb388103afdc
I20240501 19:57:28.500911 15599 raft_consensus.cc:2454] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.501480 15463 raft_consensus.cc:2454] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.501721 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c20396c4be498ebb07eb388103afdc with delete_type TABLET_DATA_DELETED (replaced by tablet 78becd64ac1d4da5b4ad1474e5f41c5e) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.501840 15064 catalog_manager.cc:2145] created HMS catalog entry for table [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:28.501897 15935 tablet_replica.cc:330] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.501953 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c20396c4be498ebb07eb388103afdc with delete_type TABLET_DATA_DELETED (replaced by tablet 78becd64ac1d4da5b4ad1474e5f41c5e) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.502012 15935 raft_consensus.cc:2229] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.502120 15936 tablet_replica.cc:330] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.502136 15059 catalog_manager.cc:6554] ReplaceTablet: tablet f7c20396c4be498ebb07eb388103afdc deleted and replaced by tablet 78becd64ac1d4da5b4ad1474e5f41c5e
I20240501 19:57:28.502209 15935 raft_consensus.cc:2258] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.502244 15936 raft_consensus.cc:2229] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.502374 15936 raft_consensus.cc:2258] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.502467 15059 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0441f7d080e84fb792251357a57aa031" } new_table_name: "default.Table_43971f18a04d42be9b5f69e32a2031ed" modify_external_catalogs: true
I20240501 19:57:28.502974 15936 ts_tablet_manager.cc:1875] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.503978 15935 ts_tablet_manager.cc:1875] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.504962 15936 ts_tablet_manager.cc:1888] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.505033 15936 log.cc:1196] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/f7c20396c4be498ebb07eb388103afdc
I20240501 19:57:28.505280 15936 ts_tablet_manager.cc:1909] T f7c20396c4be498ebb07eb388103afdc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.505815 15261 leader_election.cc:302] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.505951 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:28.506115 15935 ts_tablet_manager.cc:1888] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.506186 15935 log.cc:1196] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/f7c20396c4be498ebb07eb388103afdc
I20240501 19:57:28.506317 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet f7c20396c4be498ebb07eb388103afdc with delete_type TABLET_DATA_DELETED (replaced by tablet 78becd64ac1d4da5b4ad1474e5f41c5e) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.506422 15935 ts_tablet_manager.cc:1909] T f7c20396c4be498ebb07eb388103afdc P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.506564 15934 tablet_replica.cc:330] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.506691 15934 raft_consensus.cc:2229] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.506825 15934 raft_consensus.cc:2258] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.507001 15868 raft_consensus.cc:2781] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.507076 15868 raft_consensus.cc:686] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.507195 15868 consensus_queue.cc:237] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.507575 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:28.507910 15868 raft_consensus.cc:2781] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.507977 15868 raft_consensus.cc:686] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.508108 15868 consensus_queue.cc:237] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.509045 15741 raft_consensus.cc:482] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.509132 15741 raft_consensus.cc:504] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.509512 15741 leader_election.cc:288] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.510708 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c0ca8638f14b89a2041c10a2e2206d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:28.510828 15326 raft_consensus.cc:2454] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:28.511360 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c0ca8638f14b89a2041c10a2e2206d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.511487 15599 raft_consensus.cc:2454] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:28.511834 15934 ts_tablet_manager.cc:1875] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.511978 15397 leader_election.cc:302] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.512152 15741 raft_consensus.cc:2781] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.512204 15741 raft_consensus.cc:482] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.512239 15741 raft_consensus.cc:3037] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.513228 15741 raft_consensus.cc:504] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.514034 15934 ts_tablet_manager.cc:1888] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.514098 15934 log.cc:1196] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/f7c20396c4be498ebb07eb388103afdc
I20240501 19:57:28.513623 15741 leader_election.cc:288] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.514350 15934 ts_tablet_manager.cc:1909] T f7c20396c4be498ebb07eb388103afdc P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.514451 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c0ca8638f14b89a2041c10a2e2206d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:28.514559 15326 raft_consensus.cc:3037] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.514840 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71c0ca8638f14b89a2041c10a2e2206d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.514957 15599 raft_consensus.cc:3037] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.515426 15326 raft_consensus.cc:2454] T 71c0ca8638f14b89a2041c10a2e2206d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.515743 15397 leader_election.cc:302] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.515919 15741 raft_consensus.cc:2781] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.516003 15599 raft_consensus.cc:2454] T 71c0ca8638f14b89a2041c10a2e2206d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.516101 15741 raft_consensus.cc:686] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:28.516252 15741 consensus_queue.cc:237] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.517982 15064 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 37) took 1431 ms. Trace:
I20240501 19:57:28.518057 15064 rpcz_store.cc:276] 0501 19:57:27.086476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.251041 (+1164565us) service_pool.cc:225] Handling call
0501 19:57:28.342824 (+ 91783us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.343173 (+ 349us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.501834 (+158661us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.517787 (+ 15953us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.517889 (+ 102us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.517964 (+ 75us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.503885 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.504010 (+ 125us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.504014 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:28.504016 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:28.504085 (+ 69us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.504096 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.504099 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.504156 (+ 57us) tablet.cc:699] Row locks acquired
0501 19:57:28.504157 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:28.504190 (+ 33us) write_op.cc:270] Start()
0501 19:57:28.504214 (+ 24us) write_op.cc:276] Timestamp: P: 1714593448503779 usec, L: 0
0501 19:57:28.504216 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.504347 (+ 131us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.517297 (+ 12950us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.517401 (+ 104us) write_op.cc:301] APPLY: starting
0501 19:57:28.517419 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.517453 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.517473 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.517553 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.517585 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.517589 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.517649 (+ 60us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.517695 (+ 46us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.517711 (+ 16us) write_op.cc:446] Released schema lock
0501 19:57:28.517716 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.517772 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.517774 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":101082,"Hive Metastore.run_cpu_time_us":203,"Hive Metastore.run_wall_time_us":57425,"child_traces":[["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":409,"apply.run_wall_time_us":408,"num_ops":2,"prepare.queue_time_us":1601,"prepare.run_cpu_time_us":553,"prepare.run_wall_time_us":554,"raft.queue_time_us":1335,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":385,"replication_time_us":13057}]]}
I20240501 19:57:28.522354 15064 catalog_manager.cc:5484] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.522660 15064 catalog_manager.cc:5484] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.522836 15064 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.523311 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.523608 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78becd64ac1d4da5b4ad1474e5f41c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.525671 15906 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.525676 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.525974 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78becd64ac1d4da5b4ad1474e5f41c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.526659 15906 tablet_bootstrap.cc:654] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.528046 15906 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.528128 15906 ts_tablet_manager.cc:1367] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:28.528316 15914 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.528576 15906 raft_consensus.cc:348] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.528679 15906 raft_consensus.cc:374] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.528709 15906 raft_consensus.cc:729] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.528825 15906 consensus_queue.cc:260] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.529141 15906 ts_tablet_manager.cc:1398] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:28.529604 15914 tablet_bootstrap.cc:654] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.531210 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.531507 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78becd64ac1d4da5b4ad1474e5f41c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.532613 15914 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.532696 15914 ts_tablet_manager.cc:1367] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:28.533212 15914 raft_consensus.cc:348] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.533336 15914 raft_consensus.cc:374] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.533367 15914 raft_consensus.cc:729] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.533483 15914 consensus_queue.cc:260] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.534408 15910 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.535439 15910 tablet_bootstrap.cc:654] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.536429 15914 ts_tablet_manager.cc:1398] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:28.539973 15910 tablet_bootstrap.cc:492] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.540071 15910 ts_tablet_manager.cc:1367] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:28.540624 15910 raft_consensus.cc:348] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.540736 15910 raft_consensus.cc:374] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.540773 15910 raft_consensus.cc:729] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.541842 15868 consensus_queue.cc:1035] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.542418 15910 consensus_queue.cc:260] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.543284 15910 ts_tablet_manager.cc:1398] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:28.553078 15444 tablet_service.cc:1467] Processing CreateTablet for tablet aa19839336314324b4160d1f3c80a92d (DEFAULT_TABLE table=default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.553431 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19839336314324b4160d1f3c80a92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.555125 15058 catalog_manager.cc:2145] created HMS catalog entry for table [id=19c24eab74524422b924c3f739db4f33]
I20240501 19:57:28.555244 15906 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.556128 15906 tablet_bootstrap.cc:654] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.557994 15865 consensus_queue.cc:1035] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.559371 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet f7c20396c4be498ebb07eb388103afdc (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:28.560101 15306 tablet_service.cc:1467] Processing CreateTablet for tablet aa19839336314324b4160d1f3c80a92d (DEFAULT_TABLE table=default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.560401 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19839336314324b4160d1f3c80a92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.560595 15906 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.560683 15906 ts_tablet_manager.cc:1367] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:57:28.561183 15906 raft_consensus.cc:348] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.561296 15906 raft_consensus.cc:374] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.561334 15906 raft_consensus.cc:729] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.561458 15906 consensus_queue.cc:260] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.561755 15906 ts_tablet_manager.cc:1398] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.562507 15580 tablet_service.cc:1467] Processing CreateTablet for tablet aa19839336314324b4160d1f3c80a92d (DEFAULT_TABLE table=default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.562758 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa19839336314324b4160d1f3c80a92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.564999 15910 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.566009 15910 tablet_bootstrap.cc:654] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.567270 15058 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 32) took 1611 ms. Trace:
I20240501 19:57:28.567341 15058 rpcz_store.cc:276] 0501 19:57:26.955833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.191414 (+1235581us) service_pool.cc:225] Handling call
0501 19:57:28.345305 (+153891us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.345628 (+ 323us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.555115 (+209487us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.567046 (+ 11931us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.567155 (+ 109us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.567253 (+ 98us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.555614 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.555727 (+ 113us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.555734 (+ 7us) write_op.cc:427] Acquired schema lock
0501 19:57:28.555737 (+ 3us) tablet.cc:660] Decoding operations
0501 19:57:28.555817 (+ 80us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.555829 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.555834 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.555905 (+ 71us) tablet.cc:699] Row locks acquired
0501 19:57:28.555908 (+ 3us) write_op.cc:260] PREPARE: finished
0501 19:57:28.555941 (+ 33us) write_op.cc:270] Start()
0501 19:57:28.555968 (+ 27us) write_op.cc:276] Timestamp: P: 1714593448555525 usec, L: 0
0501 19:57:28.555972 (+ 4us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.556127 (+ 155us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.566172 (+ 10045us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.566289 (+ 117us) write_op.cc:301] APPLY: starting
0501 19:57:28.566305 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.566339 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.566355 (+ 16us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.566430 (+ 75us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.566460 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.566464 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.566528 (+ 64us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.566588 (+ 60us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.566605 (+ 17us) write_op.cc:446] Released schema lock
0501 19:57:28.566611 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.566660 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.566662 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":156103,"Hive Metastore.run_cpu_time_us":221,"Hive Metastore.run_wall_time_us":53152,"child_traces":[["op",{"apply.queue_time_us":66,"apply.run_cpu_time_us":421,"apply.run_wall_time_us":421,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":591,"prepare.run_wall_time_us":592,"replication_time_us":10175}]]}
I20240501 19:57:28.567548 15064 catalog_manager.cc:5484] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.568244 15058 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
W20240501 19:57:28.568550 15058 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 78becd64ac1d4da5b4ad1474e5f41c5e consensus state has no leader
I20240501 19:57:28.569236 15058 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.569674 15058 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5" } new_table_name: "default.Table_557eeb62e4e64cc6aeca0b1882dd3909" modify_external_catalogs: true
I20240501 19:57:28.584149 15914 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.585316 15914 tablet_bootstrap.cc:654] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.586165 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (DEFAULT_TABLE table=default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.586444 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.586524 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (DEFAULT_TABLE table=default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.586798 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.590008 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (DEFAULT_TABLE table=default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.590337 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.592959 15906 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.594180 15906 tablet_bootstrap.cc:654] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.594519 15064 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.595332 15906 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.595417 15906 ts_tablet_manager.cc:1367] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:28.595885 15906 raft_consensus.cc:348] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.596000 15906 raft_consensus.cc:374] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.596041 15906 raft_consensus.cc:729] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.596197 15906 consensus_queue.cc:260] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.596518 15906 ts_tablet_manager.cc:1398] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.598187 15064 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.599440 15064 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0c593340e8374d2d809dda0943eb4dd7" } new_table_name: "default.Table_5481ef3ce5fa4ea2b569d6ac2a569b75" modify_external_catalogs: true
I20240501 19:57:28.600447 15914 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.600523 15910 tablet_bootstrap.cc:492] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.600601 15910 ts_tablet_manager.cc:1367] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20240501 19:57:28.600557 15914 ts_tablet_manager.cc:1367] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:57:28.601212 15910 raft_consensus.cc:348] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.601217 15914 raft_consensus.cc:348] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.601322 15910 raft_consensus.cc:374] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.601322 15914 raft_consensus.cc:374] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.601356 15914 raft_consensus.cc:729] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.601356 15910 raft_consensus.cc:729] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.601478 15910 consensus_queue.cc:260] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.601478 15914 consensus_queue.cc:260] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.601744 15910 ts_tablet_manager.cc:1398] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.601744 15914 ts_tablet_manager.cc:1398] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.601926 15914 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.601933 15910 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.602890 15914 tablet_bootstrap.cc:654] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.602890 15910 tablet_bootstrap.cc:654] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.603937 15055 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]
I20240501 19:57:28.606628 15598 raft_consensus.cc:1261] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.607709 15725 consensus_queue.cc:1035] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.607883 15463 raft_consensus.cc:1261] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Refusing update from remote peer 0c8dfc11dc7542f583644b9ac747545d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.608280 15725 consensus_queue.cc:1035] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.614116 15910 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.614123 15920 tablet.cc:1722] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.614228 15910 ts_tablet_manager.cc:1367] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:28.614997 15868 raft_consensus.cc:482] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.615075 15868 raft_consensus.cc:504] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.615298 15910 raft_consensus.cc:348] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.615420 15910 raft_consensus.cc:374] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.615430 15868 leader_election.cc:288] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.615463 15910 raft_consensus.cc:729] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.615590 15910 consensus_queue.cc:260] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.615880 15910 ts_tablet_manager.cc:1398] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:28.616674 15723 raft_consensus.cc:482] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.616756 15723 raft_consensus.cc:504] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.617152 15723 leader_election.cc:288] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.617434 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:28.617547 15326 raft_consensus.cc:2454] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:28.617638 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.617750 15599 raft_consensus.cc:2454] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.617820 15533 leader_election.cc:302] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.617964 15723 raft_consensus.cc:2781] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.618010 15723 raft_consensus.cc:482] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.618043 15723 raft_consensus.cc:3037] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.618984 15723 raft_consensus.cc:504] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.619309 15723 leader_election.cc:288] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.619930 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.620066 15462 raft_consensus.cc:3037] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.620323 15461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.620676 15918 tablet.cc:1722] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.621058 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.621127 15462 raft_consensus.cc:2454] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:28.622561 15261 leader_election.cc:302] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.623922 15865 raft_consensus.cc:2781] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.623986 15865 raft_consensus.cc:482] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.624023 15865 raft_consensus.cc:3037] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.625018 15865 raft_consensus.cc:504] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.625396 15865 leader_election.cc:288] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.625903 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:28.626051 15326 raft_consensus.cc:2379] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f011d422e164c949a9d54d846b7c432 in current term 1: Already voted for candidate 0c8dfc11dc7542f583644b9ac747545d in this term.
I20240501 19:57:28.626474 15536 leader_election.cc:302] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.627707 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.627895 15463 raft_consensus.cc:2379] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c8dfc11dc7542f583644b9ac747545d in current term 1: Already voted for candidate 7f011d422e164c949a9d54d846b7c432 in this term.
I20240501 19:57:28.628926 15914 tablet_bootstrap.cc:492] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.629022 15914 ts_tablet_manager.cc:1367] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 19:57:28.629473 15914 raft_consensus.cc:348] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.629550 15914 raft_consensus.cc:374] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.629571 15914 raft_consensus.cc:729] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.629675 15914 consensus_queue.cc:260] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.629936 15914 ts_tablet_manager.cc:1398] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:28.631500 15868 raft_consensus.cc:482] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.631584 15868 raft_consensus.cc:504] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.631848 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78becd64ac1d4da5b4ad1474e5f41c5e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.631958 15868 leader_election.cc:288] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.632007 15598 raft_consensus.cc:2379] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0c8dfc11dc7542f583644b9ac747545d in current term 1: Already voted for candidate 7f011d422e164c949a9d54d846b7c432 in this term.
I20240501 19:57:28.632398 15968 raft_consensus.cc:482] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.632488 15968 raft_consensus.cc:504] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.632900 15968 leader_election.cc:288] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.633503 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed5f03a8ff34913a59c475bd7ec83cf" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.633625 15598 raft_consensus.cc:2454] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:28.633919 15399 leader_election.cc:302] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.634291 15968 raft_consensus.cc:2781] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.634346 15968 raft_consensus.cc:482] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.634382 15968 raft_consensus.cc:3037] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.634902 15723 raft_consensus.cc:2781] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.634999 15723 raft_consensus.cc:686] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:28.635143 15723 consensus_queue.cc:237] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.635486 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6cb71ea54e4cc2985b9d607109aba5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.635609 15463 raft_consensus.cc:2454] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.635980 15968 raft_consensus.cc:504] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.636337 15968 leader_election.cc:288] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.636740 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed5f03a8ff34913a59c475bd7ec83cf" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.636873 15598 raft_consensus.cc:3037] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.637640 15262 leader_election.cc:302] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.637848 15598 raft_consensus.cc:2454] T 2ed5f03a8ff34913a59c475bd7ec83cf P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.638226 15725 raft_consensus.cc:2781] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.638304 15725 raft_consensus.cc:482] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.638348 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed5f03a8ff34913a59c475bd7ec83cf" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:28.638489 15325 raft_consensus.cc:3037] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.639007 15932 tablet.cc:1722] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.639387 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed5f03a8ff34913a59c475bd7ec83cf" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:28.639456 15325 raft_consensus.cc:2454] T 2ed5f03a8ff34913a59c475bd7ec83cf P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.638351 15725 raft_consensus.cc:3037] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.639889 15399 leader_election.cc:302] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.640668 15725 raft_consensus.cc:504] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.638371 15261 leader_election.cc:302] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d; no voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:28.642616 15868 raft_consensus.cc:2726] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 19:57:28.643797 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6cb71ea54e4cc2985b9d607109aba5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.643924 15598 raft_consensus.cc:2454] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.644275 15725 leader_election.cc:288] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.644690 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6cb71ea54e4cc2985b9d607109aba5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.644857 15463 raft_consensus.cc:3037] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.644860 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6cb71ea54e4cc2985b9d607109aba5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.645062 15598 raft_consensus.cc:3037] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.645926 15463 raft_consensus.cc:2454] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.646040 15598 raft_consensus.cc:2454] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.646276 15262 leader_election.cc:302] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.646502 15725 raft_consensus.cc:2781] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.646601 15725 raft_consensus.cc:686] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.646752 15725 consensus_queue.cc:237] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.647593 15968 raft_consensus.cc:2781] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.647682 15968 raft_consensus.cc:686] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:28.647804 15968 consensus_queue.cc:237] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.650394 15062 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]
I20240501 19:57:28.687963 15060 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]
I20240501 19:57:28.703649 15723 raft_consensus.cc:482] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.703796 15723 raft_consensus.cc:504] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.704274 15723 leader_election.cc:288] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.704946 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19839336314324b4160d1f3c80a92d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:28.705094 15325 raft_consensus.cc:2454] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:28.705456 15533 leader_election.cc:302] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.705955 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19839336314324b4160d1f3c80a92d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.706106 15463 raft_consensus.cc:2454] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:28.706866 15723 raft_consensus.cc:2781] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.706935 15723 raft_consensus.cc:482] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.706980 15723 raft_consensus.cc:3037] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.708072 15723 raft_consensus.cc:504] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.708477 15723 leader_election.cc:288] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:28.709175 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19839336314324b4160d1f3c80a92d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:28.709306 15325 raft_consensus.cc:3037] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.710318 15325 raft_consensus.cc:2454] T aa19839336314324b4160d1f3c80a92d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:28.710652 15533 leader_election.cc:302] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.711020 15723 raft_consensus.cc:2781] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.711071 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa19839336314324b4160d1f3c80a92d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.711108 15723 raft_consensus.cc:686] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:28.711194 15463 raft_consensus.cc:3037] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.711287 15723 consensus_queue.cc:237] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.711632 15056 catalog_manager.cc:2145] created HMS catalog entry for table [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:28.718243 15056 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 39) took 1489 ms. Trace:
I20240501 19:57:28.718333 15056 rpcz_store.cc:276] 0501 19:57:27.228311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.317258 (+1088947us) service_pool.cc:225] Handling call
0501 19:57:28.396089 (+ 78831us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.396455 (+ 366us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.711624 (+315169us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.718031 (+ 6407us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.718130 (+ 99us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.718223 (+ 93us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.712723 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.712830 (+ 107us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.712855 (+ 25us) write_op.cc:427] Acquired schema lock
0501 19:57:28.712858 (+ 3us) tablet.cc:660] Decoding operations
0501 19:57:28.712938 (+ 80us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.712949 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.712952 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.713018 (+ 66us) tablet.cc:699] Row locks acquired
0501 19:57:28.713020 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:28.713049 (+ 29us) write_op.cc:270] Start()
0501 19:57:28.713080 (+ 31us) write_op.cc:276] Timestamp: P: 1714593448712620 usec, L: 0
0501 19:57:28.713082 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.713219 (+ 137us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.716805 (+ 3586us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.717410 (+ 605us) write_op.cc:301] APPLY: starting
0501 19:57:28.717428 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.717461 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.717476 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.717570 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.717600 (+ 30us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.717605 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:28.717666 (+ 61us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.717717 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.717731 (+ 14us) write_op.cc:446] Released schema lock
0501 19:57:28.717735 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.717784 (+ 49us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.717787 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":291418,"Hive Metastore.run_cpu_time_us":217,"Hive Metastore.run_wall_time_us":23442,"child_traces":[["op",{"apply.queue_time_us":553,"apply.run_cpu_time_us":421,"apply.run_wall_time_us":421,"num_ops":2,"prepare.queue_time_us":609,"prepare.run_cpu_time_us":585,"prepare.run_wall_time_us":586,"raft.queue_time_us":359,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":355,"replication_time_us":3703}]]}
I20240501 19:57:28.728574 15056 catalog_manager.cc:5484] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.728992 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.729291 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e2ff9f8a4df2b0856b9041dd2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.729413 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.729712 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e2ff9f8a4df2b0856b9041dd2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.729826 15463 raft_consensus.cc:2454] T aa19839336314324b4160d1f3c80a92d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:28.729933 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.730187 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d85e2ff9f8a4df2b0856b9041dd2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.731350 15906 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.731873 15914 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.732277 15910 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.732317 15906 tablet_bootstrap.cc:654] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.732744 15914 tablet_bootstrap.cc:654] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.733211 15910 tablet_bootstrap.cc:654] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.733476 15906 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.733574 15906 ts_tablet_manager.cc:1367] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:28.734071 15906 raft_consensus.cc:348] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.734246 15906 raft_consensus.cc:374] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.734289 15906 raft_consensus.cc:729] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.734426 15906 consensus_queue.cc:260] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.736071 15906 ts_tablet_manager.cc:1398] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:28.739221 15056 catalog_manager.cc:5484] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.743863 15056 catalog_manager.cc:5484] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:28.744171 15056 catalog_manager.cc:5484] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:28.744313 15056 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.749554 15056 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_ea70d92de3fc40b4b3e8c54d6e5b6899" } new_table_name: "default.Table_9254e341432a42c29665f212b2913ff6" modify_external_catalogs: true
I20240501 19:57:28.757340 15914 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.757469 15914 ts_tablet_manager.cc:1367] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 19:57:28.757980 15914 raft_consensus.cc:348] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.758081 15910 tablet_bootstrap.cc:492] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.758107 15914 raft_consensus.cc:374] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.758167 15914 raft_consensus.cc:729] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.758201 15910 ts_tablet_manager.cc:1367] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.000s
I20240501 19:57:28.758296 15914 consensus_queue.cc:260] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.758579 15914 ts_tablet_manager.cc:1398] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.758682 15910 raft_consensus.cc:348] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.758806 15910 raft_consensus.cc:374] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.758843 15910 raft_consensus.cc:729] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.759021 15910 consensus_queue.cc:260] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.759311 15910 ts_tablet_manager.cc:1398] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.765105 15325 raft_consensus.cc:1261] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Refusing update from remote peer 7f011d422e164c949a9d54d846b7c432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.765537 15463 raft_consensus.cc:1261] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Refusing update from remote peer 7f011d422e164c949a9d54d846b7c432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:28.766086 15752 consensus_queue.cc:1035] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.770893 15752 consensus_queue.cc:1035] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.771440 15063 catalog_manager.cc:2145] created HMS catalog entry for table [id=cb06dd07345741c8a5bd80df323d4f96]
I20240501 19:57:28.774055 15932 tablet.cc:1722] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.774910 15865 raft_consensus.cc:482] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.775024 15865 raft_consensus.cc:504] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.775420 15865 leader_election.cc:288] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.776039 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561867ecbf3c4f3a9b28f91f38ac1f4b" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:28.776167 15462 raft_consensus.cc:2454] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.776683 15262 leader_election.cc:302] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.777287 15918 tablet.cc:1722] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.777315 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561867ecbf3c4f3a9b28f91f38ac1f4b" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.777459 15598 raft_consensus.cc:2454] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:28.779167 15063 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 38) took 1604 ms. Trace:
I20240501 19:57:28.779239 15063 rpcz_store.cc:276] 0501 19:57:27.174676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.269895 (+1095219us) service_pool.cc:225] Handling call
0501 19:57:28.396445 (+126550us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.396761 (+ 316us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.771431 (+374670us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.778976 (+ 7545us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.779073 (+ 97us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.779151 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.771964 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.772064 (+ 100us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.772068 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:28.772070 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:28.772135 (+ 65us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.772148 (+ 13us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.772150 (+ 2us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.772216 (+ 66us) tablet.cc:699] Row locks acquired
0501 19:57:28.772217 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:28.772244 (+ 27us) write_op.cc:270] Start()
0501 19:57:28.772269 (+ 25us) write_op.cc:276] Timestamp: P: 1714593448771812 usec, L: 0
0501 19:57:28.772273 (+ 4us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.772405 (+ 132us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.775030 (+ 2625us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x5597cdb9bac8. stack: 00007fb4d7fc6903 00007fb4d7fc6ef5 00007fb4d7b4889d 00005597cc60a36e 00005597cc60a3c1 00005597cc60e6fa 00007fb4d9b40765 00007fb4d9b4099b 00007fb4d9b115d1 00007fb4d9b00156 00007fb4d9affce9 00007fb4d9b049ac 00005597cc69975b 00007fb4d7fe9e05 00007fb4d7fea6ee 00007fb4d7fec03a
0501 19:57:28.777695 (+ 2665us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.777788 (+ 93us) write_op.cc:301] APPLY: starting
0501 19:57:28.777805 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.777835 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.777836 (+ 1us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.777917 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.777950 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.777954 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.778009 (+ 55us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.778055 (+ 46us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.778071 (+ 16us) write_op.cc:446] Released schema lock
0501 19:57:28.778075 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.778123 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.778125 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":314627,"Hive Metastore.run_cpu_time_us":187,"Hive Metastore.run_wall_time_us":59918,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":384,"apply.run_wall_time_us":384,"num_ops":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":524,"prepare.run_wall_time_us":524,"raft.queue_time_us":745,"raft.run_cpu_time_us":791,"raft.run_wall_time_us":2740,"replication_time_us":5403,"spinlock_wait_cycles":4396416}]]}
I20240501 19:57:28.781080 15865 raft_consensus.cc:2781] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.781155 15865 raft_consensus.cc:482] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.781198 15865 raft_consensus.cc:3037] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.781745 15063 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.782243 15865 raft_consensus.cc:504] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.782625 15865 leader_election.cc:288] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.783317 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561867ecbf3c4f3a9b28f91f38ac1f4b" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:28.784425 15462 raft_consensus.cc:3037] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.785745 15462 raft_consensus.cc:2454] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.786666 15210 catalog_manager.cc:2706] Deleting table default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4] in response to Hive Metastore notification log event 25
I20240501 19:57:28.787592 15262 leader_election.cc:302] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.790212 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561867ecbf3c4f3a9b28f91f38ac1f4b" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.790355 15598 raft_consensus.cc:3037] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.791010 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 34f6808094c74a4e8b3b7d092d863042 (DEFAULT_TABLE table=default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.791326 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f6808094c74a4e8b3b7d092d863042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.791412 15598 raft_consensus.cc:2454] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:28.793085 15865 raft_consensus.cc:2781] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.793184 15865 raft_consensus.cc:686] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:28.793319 15865 consensus_queue.cc:237] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.793998 15914 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.794961 15914 tablet_bootstrap.cc:654] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.795310 15063 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:28.795828 15063 master_service.cc:946] ReplaceTablet: received request to replace tablet 0e3f46ab4e72478ca29bfca14391db07 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.795930 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 34f6808094c74a4e8b3b7d092d863042 (DEFAULT_TABLE table=default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.796200 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f6808094c74a4e8b3b7d092d863042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.798038 15210 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet e79d98a2beff4dc690b39a0a5ac04ded
I20240501 19:57:28.798671 15210 catalog_manager.cc:2706] Deleting table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad] in response to Hive Metastore notification log event 26
I20240501 19:57:28.799162 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.799337 15936 tablet_replica.cc:330] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.799461 15936 raft_consensus.cc:2229] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.799566 15936 raft_consensus.cc:2258] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.799921 15936 ts_tablet_manager.cc:1875] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.801352 15920 tablet.cc:1722] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:28.802078 15936 ts_tablet_manager.cc:1888] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.802150 15936 log.cc:1196] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/e79d98a2beff4dc690b39a0a5ac04ded
I20240501 19:57:28.802394 15936 ts_tablet_manager.cc:1909] T e79d98a2beff4dc690b39a0a5ac04ded P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.803309 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 34f6808094c74a4e8b3b7d092d863042 (DEFAULT_TABLE table=default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.803575 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f6808094c74a4e8b3b7d092d863042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.804257 15063 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 0e3f46ab4e72478ca29bfca14391db07
I20240501 19:57:28.805197 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.805357 15935 tablet_replica.cc:330] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.805483 15935 raft_consensus.cc:2229] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.805667 15935 raft_consensus.cc:2258] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.806047 15935 ts_tablet_manager.cc:1875] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.808005 15935 ts_tablet_manager.cc:1888] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.808064 15935 log.cc:1196] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/e79d98a2beff4dc690b39a0a5ac04ded
I20240501 19:57:28.808313 15935 ts_tablet_manager.cc:1909] T e79d98a2beff4dc690b39a0a5ac04ded P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.812003 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 with delete_type TABLET_DATA_DELETED (replaced by tablet 9384878f4f834c9d90db68234d8755ae) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.812475 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 with delete_type TABLET_DATA_DELETED (replaced by tablet 9384878f4f834c9d90db68234d8755ae) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.812649 15934 tablet_replica.cc:330] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.812773 15934 raft_consensus.cc:2229] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.812922 15934 raft_consensus.cc:2258] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.813014 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet e79d98a2beff4dc690b39a0a5ac04ded (table default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]) successfully deleted
I20240501 19:57:28.813304 15934 ts_tablet_manager.cc:1875] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.813313 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e3f46ab4e72478ca29bfca14391db07 with delete_type TABLET_DATA_DELETED (replaced by tablet 9384878f4f834c9d90db68234d8755ae) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.814941 15063 catalog_manager.cc:6554] ReplaceTablet: tablet 0e3f46ab4e72478ca29bfca14391db07 deleted and replaced by tablet 9384878f4f834c9d90db68234d8755ae
I20240501 19:57:28.815140 15210 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 9384878f4f834c9d90db68234d8755ae
I20240501 19:57:28.815353 15934 ts_tablet_manager.cc:1888] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.815425 15934 log.cc:1196] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/0e3f46ab4e72478ca29bfca14391db07
I20240501 19:57:28.815667 15934 ts_tablet_manager.cc:1909] T 0e3f46ab4e72478ca29bfca14391db07 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.815698 15906 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.815824 15210 catalog_manager.cc:2706] Deleting table default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332] in response to Hive Metastore notification log event 27
I20240501 19:57:28.816645 15906 tablet_bootstrap.cc:654] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.816610 15063 catalog_manager.cc:5484] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.817561 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.818228 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.818516 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 9384878f4f834c9d90db68234d8755ae (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.818753 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9384878f4f834c9d90db68234d8755ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.818818 15906 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.818881 15906 ts_tablet_manager.cc:1367] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:28.819247 15906 raft_consensus.cc:348] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.819324 15906 raft_consensus.cc:374] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.819342 15906 raft_consensus.cc:729] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.819454 15906 consensus_queue.cc:260] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.819723 15906 ts_tablet_manager.cc:1398] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:28.820622 15210 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 4814dbbccd08455aa0b7381de1a34616
I20240501 19:57:28.821362 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.821642 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 9384878f4f834c9d90db68234d8755ae (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.821885 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9384878f4f834c9d90db68234d8755ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.824393 15910 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.825400 15910 tablet_bootstrap.cc:654] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.827158 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814dbbccd08455aa0b7381de1a34616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.829511 15063 master_service.cc:946] ReplaceTablet: received request to replace tablet ccdc0cb3c51e4f87bc9f1d1a03984591 from {username='slave'} at 127.0.0.1:36756
W20240501 19:57:28.830250 15061 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_090e62cd0c6c43eb9c44ada659e01c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.833407 15063 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet ccdc0cb3c51e4f87bc9f1d1a03984591
I20240501 19:57:28.833695 15063 catalog_manager.cc:6554] ReplaceTablet: tablet ccdc0cb3c51e4f87bc9f1d1a03984591 deleted and replaced by tablet 2a049fb6ddfd4cf49571cfbe9f4375a1
I20240501 19:57:28.833859 15063 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_b0d6857034b54b258f3f6d577cd53e13" } new_table_name: "default.Table_0e2de9c19f564d149e60a8bac7d8ae48" modify_external_catalogs: true
I20240501 19:57:28.835456 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a049fb6ddfd4cf49571cfbe9f4375a1) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.836253 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a049fb6ddfd4cf49571cfbe9f4375a1) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.836302 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet ccdc0cb3c51e4f87bc9f1d1a03984591 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a049fb6ddfd4cf49571cfbe9f4375a1) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.838299 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814dbbccd08455aa0b7381de1a34616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.838824 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 9384878f4f834c9d90db68234d8755ae (DEFAULT_TABLE table=default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.839121 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9384878f4f834c9d90db68234d8755ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.840131 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.840312 15442 tablet_service.cc:1467] Processing CreateTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.840409 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.840524 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.844045 15906 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.844051 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.844300 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.845890 15906 tablet_bootstrap.cc:654] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.846781 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet e79d98a2beff4dc690b39a0a5ac04ded with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.846964 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 4814dbbccd08455aa0b7381de1a34616 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:28 UTC) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.847673 15934 tablet_replica.cc:330] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
W20240501 19:57:28.847766 15048 catalog_manager.cc:4810] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): delete failed for tablet 9384878f4f834c9d90db68234d8755ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9384878f4f834c9d90db68234d8755ae already in progress: creating tablet
I20240501 19:57:28.847817 15934 raft_consensus.cc:2229] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.848011 15934 raft_consensus.cc:2258] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.848623 15936 tablet_replica.cc:330] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.848666 15934 ts_tablet_manager.cc:1875] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.848747 15936 raft_consensus.cc:2229] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.848865 15936 raft_consensus.cc:2258] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.848773 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet e79d98a2beff4dc690b39a0a5ac04ded (table default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]) successfully deleted
I20240501 19:57:28.849213 15936 ts_tablet_manager.cc:1875] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.850833 15934 ts_tablet_manager.cc:1888] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.850898 15934 log.cc:1196] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/ccdc0cb3c51e4f87bc9f1d1a03984591
I20240501 19:57:28.851145 15934 ts_tablet_manager.cc:1909] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.851300 15936 ts_tablet_manager.cc:1888] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.851365 15936 log.cc:1196] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/0e3f46ab4e72478ca29bfca14391db07
I20240501 19:57:28.851583 15936 ts_tablet_manager.cc:1909] T 0e3f46ab4e72478ca29bfca14391db07 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.851847 15935 tablet_replica.cc:330] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
W20240501 19:57:28.851966 15536 consensus_peers.cc:485] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 -> Peer 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): Couldn't send request to peer 68539c0ecd3e44e080cb53987d4f4c23. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:28.851994 15935 raft_consensus.cc:2229] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.852221 15935 raft_consensus.cc:2258] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.852604 15935 ts_tablet_manager.cc:1875] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.852676 15934 tablet_replica.cc:330] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.852648 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.852792 15934 raft_consensus.cc:2229] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.853003 15934 raft_consensus.cc:2258] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.853317 15934 ts_tablet_manager.cc:1875] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 19:57:28.853721 15051 catalog_manager.cc:4810] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): delete failed for tablet 9384878f4f834c9d90db68234d8755ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9384878f4f834c9d90db68234d8755ae already in progress: creating tablet
I20240501 19:57:28.853972 15936 tablet_replica.cc:330] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.854135 15936 raft_consensus.cc:2229] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.854306 15936 raft_consensus.cc:2258] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.854723 15935 ts_tablet_manager.cc:1888] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.854724 15936 ts_tablet_manager.cc:1875] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.854799 15935 log.cc:1196] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/0e3f46ab4e72478ca29bfca14391db07
I20240501 19:57:28.855057 15935 ts_tablet_manager.cc:1909] T 0e3f46ab4e72478ca29bfca14391db07 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.855304 15934 ts_tablet_manager.cc:1888] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.855383 15934 log.cc:1196] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/4814dbbccd08455aa0b7381de1a34616
I20240501 19:57:28.855624 15934 ts_tablet_manager.cc:1909] T 4814dbbccd08455aa0b7381de1a34616 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
W20240501 19:57:28.856204 15050 catalog_manager.cc:4810] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): delete failed for tablet 9384878f4f834c9d90db68234d8755ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9384878f4f834c9d90db68234d8755ae already in progress: creating tablet
I20240501 19:57:28.856215 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 4814dbbccd08455aa0b7381de1a34616 (table default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]) successfully deleted
I20240501 19:57:28.856429 15934 tablet_replica.cc:330] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.856429 15935 tablet_replica.cc:330] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.856549 15934 raft_consensus.cc:2229] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.856554 15935 raft_consensus.cc:2229] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.856652 15934 raft_consensus.cc:2258] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.856662 15935 raft_consensus.cc:2258] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.857271 15935 ts_tablet_manager.cc:1875] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.857496 15936 ts_tablet_manager.cc:1888] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.857625 15936 log.cc:1196] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/4814dbbccd08455aa0b7381de1a34616
I20240501 19:57:28.857860 15910 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.857972 15910 ts_tablet_manager.cc:1367] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.002s
I20240501 19:57:28.858494 15910 raft_consensus.cc:348] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.858603 15910 raft_consensus.cc:374] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.858633 15910 raft_consensus.cc:729] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.858760 15910 consensus_queue.cc:260] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.859076 15910 ts_tablet_manager.cc:1398] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:28.859163 15935 ts_tablet_manager.cc:1888] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.859258 15935 log.cc:1196] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/ccdc0cb3c51e4f87bc9f1d1a03984591
I20240501 19:57:28.859303 15910 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.859540 15935 ts_tablet_manager.cc:1909] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.860049 15935 tablet_replica.cc:330] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.860155 15935 raft_consensus.cc:2229] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.860458 15936 ts_tablet_manager.cc:1909] T 4814dbbccd08455aa0b7381de1a34616 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.860599 15934 ts_tablet_manager.cc:1875] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.861269 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 4814dbbccd08455aa0b7381de1a34616 (table default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]) successfully deleted
I20240501 19:57:28.862244 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.862593 15936 tablet_replica.cc:330] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.862618 15934 ts_tablet_manager.cc:1888] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.862689 15934 log.cc:1196] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/e79d98a2beff4dc690b39a0a5ac04ded
I20240501 19:57:28.862834 15936 raft_consensus.cc:2229] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.862946 15934 ts_tablet_manager.cc:1909] T e79d98a2beff4dc690b39a0a5ac04ded P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.863039 15936 raft_consensus.cc:2258] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.863219 15906 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.863304 15906 ts_tablet_manager.cc:1367] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240501 19:57:28.860252 15935 raft_consensus.cc:2258] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.863691 15936 ts_tablet_manager.cc:1875] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.863756 15935 ts_tablet_manager.cc:1875] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.863776 15906 raft_consensus.cc:348] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.863894 15906 raft_consensus.cc:374] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.863938 15906 raft_consensus.cc:729] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.860312 15910 tablet_bootstrap.cc:654] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.864668 15906 consensus_queue.cc:260] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.865734 15935 ts_tablet_manager.cc:1888] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.865808 15935 log.cc:1196] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/4814dbbccd08455aa0b7381de1a34616
I20240501 19:57:28.866056 15935 ts_tablet_manager.cc:1909] T 4814dbbccd08455aa0b7381de1a34616 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.866696 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet e79d98a2beff4dc690b39a0a5ac04ded (table default.table_0ca797a676d54ea89cfea3963a75b1d0 [id=f95181efc2104a7ab9796cc0e11a64e4]) successfully deleted
I20240501 19:57:28.867300 15936 ts_tablet_manager.cc:1888] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:28.867432 15936 log.cc:1196] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/ccdc0cb3c51e4f87bc9f1d1a03984591
I20240501 19:57:28.867467 15906 ts_tablet_manager.cc:1398] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:57:28.867487 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 4814dbbccd08455aa0b7381de1a34616 (table default.table_31c7944e4b41499683c266b366d0711f [id=076667aca4134cf9a588a3dc30297332]) successfully deleted
I20240501 19:57:28.867784 15936 ts_tablet_manager.cc:1909] T ccdc0cb3c51e4f87bc9f1d1a03984591 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.868482 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet ccdc0cb3c51e4f87bc9f1d1a03984591 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:28.869014 15906 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.869937 15906 tablet_bootstrap.cc:654] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.870968 15906 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.870955 15914 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.871047 15914 ts_tablet_manager.cc:1367] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.000s
I20240501 19:57:28.871044 15906 ts_tablet_manager.cc:1367] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:28.871519 15906 raft_consensus.cc:348] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.871500 15914 raft_consensus.cc:348] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.871618 15914 raft_consensus.cc:374] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.871618 15906 raft_consensus.cc:374] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.871651 15914 raft_consensus.cc:729] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.871651 15906 raft_consensus.cc:729] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.871779 15914 consensus_queue.cc:260] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.871806 15906 consensus_queue.cc:260] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.872067 15914 ts_tablet_manager.cc:1398] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.872249 15914 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.873270 15914 tablet_bootstrap.cc:654] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.873643 15906 ts_tablet_manager.cc:1398] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:57:28.874295 15910 tablet_bootstrap.cc:492] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.874389 15910 ts_tablet_manager.cc:1367] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.002s
I20240501 19:57:28.874477 15741 raft_consensus.cc:482] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:28.874569 15741 raft_consensus.cc:504] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.874861 15910 raft_consensus.cc:348] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.874934 15741 leader_election.cc:288] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.874972 15910 raft_consensus.cc:374] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.875005 15910 raft_consensus.cc:729] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.875130 15910 consensus_queue.cc:260] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.875429 15910 ts_tablet_manager.cc:1398] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:28.875622 15910 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.875702 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e2ff9f8a4df2b0856b9041dd2919" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:28.875797 15598 raft_consensus.cc:2454] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:28.876078 15399 leader_election.cc:302] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:28.876438 15741 raft_consensus.cc:2781] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:28.876493 15741 raft_consensus.cc:482] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:28.876542 15741 raft_consensus.cc:3037] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.877210 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e2ff9f8a4df2b0856b9041dd2919" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:28.877332 15325 raft_consensus.cc:2454] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:28.877559 15741 raft_consensus.cc:504] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.877946 15741 leader_election.cc:288] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:28.878544 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e2ff9f8a4df2b0856b9041dd2919" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:28.878641 15598 raft_consensus.cc:3037] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.879613 15598 raft_consensus.cc:2454] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.879932 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d85e2ff9f8a4df2b0856b9041dd2919" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:28.880048 15325 raft_consensus.cc:3037] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:28.880438 15914 tablet_bootstrap.cc:492] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.880524 15914 ts_tablet_manager.cc:1367] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:28.880559 15910 tablet_bootstrap.cc:654] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.881002 15325 raft_consensus.cc:2454] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:28.881103 15914 raft_consensus.cc:348] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.881206 15914 raft_consensus.cc:374] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.881246 15914 raft_consensus.cc:729] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.881402 15397 leader_election.cc:302] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:28.881377 15914 consensus_queue.cc:260] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.881582 15741 raft_consensus.cc:2781] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:28.881649 15741 raft_consensus.cc:686] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:28.881700 15914 ts_tablet_manager.cc:1398] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.881764 15741 consensus_queue.cc:237] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.881860 15910 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.881870 15914 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.881937 15910 ts_tablet_manager.cc:1367] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:28.882402 15910 raft_consensus.cc:348] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.882526 15910 raft_consensus.cc:374] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.882566 15910 raft_consensus.cc:729] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.882684 15910 consensus_queue.cc:260] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.882973 15910 ts_tablet_manager.cc:1398] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.883023 15914 tablet_bootstrap.cc:654] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.885207 15061 catalog_manager.cc:2145] created HMS catalog entry for table [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:28.889732 15061 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36756 (request call id 40) took 1625 ms. Trace:
I20240501 19:57:28.889829 15061 rpcz_store.cc:276] 0501 19:57:27.264612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.411505 (+1146893us) service_pool.cc:225] Handling call
0501 19:57:28.830333 (+418828us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:28.830649 (+ 316us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:28.885197 (+ 54548us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:28.889552 (+ 4355us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:28.889643 (+ 91us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:28.889716 (+ 73us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:28.885741 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:28.885842 (+ 101us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:28.885846 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:28.885848 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:28.885924 (+ 76us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:28.885935 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:28.885938 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:28.886000 (+ 62us) tablet.cc:699] Row locks acquired
0501 19:57:28.886003 (+ 3us) write_op.cc:260] PREPARE: finished
0501 19:57:28.886034 (+ 31us) write_op.cc:270] Start()
0501 19:57:28.886063 (+ 29us) write_op.cc:276] Timestamp: P: 1714593448885590 usec, L: 0
0501 19:57:28.886066 (+ 3us) op_driver.cc:331] REPLICATION: starting
0501 19:57:28.886211 (+ 145us) log.cc:842] Serialized 432 byte log entry
0501 19:57:28.888806 (+ 2595us) op_driver.cc:444] REPLICATION: finished
0501 19:57:28.889218 (+ 412us) write_op.cc:301] APPLY: starting
0501 19:57:28.889234 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:28.889266 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:28.889280 (+ 14us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:28.889353 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:28.889382 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:28.889386 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:28.889438 (+ 52us) log.cc:842] Serialized 28 byte log entry
0501 19:57:28.889477 (+ 39us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.889486 (+ 9us) write_op.cc:446] Released schema lock
0501 19:57:28.889490 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:28.889542 (+ 52us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:28.889545 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":27,"Hive Metastore.run_cpu_time_us":205,"Hive Metastore.run_wall_time_us":54270,"child_traces":[["op",{"apply.queue_time_us":368,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":367,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":567,"prepare.run_wall_time_us":568,"raft.queue_time_us":873,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":304,"replication_time_us":2722}]]}
I20240501 19:57:28.892632 15061 master_service.cc:946] ReplaceTablet: received request to replace tablet 539033016933451fa52037d581015148 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.893584 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.895295 15060 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36756 (request call id 36) took 1831 ms. Trace:
I20240501 19:57:28.895362 15060 rpcz_store.cc:276] 0501 19:57:27.063562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.233696 (+1170134us) service_pool.cc:225] Handling call
0501 19:57:28.350405 (+116709us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.895282 (+544877us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":299702,"Hive Metastore.run_cpu_time_us":189,"Hive Metastore.run_wall_time_us":37697}
W20240501 19:57:28.895483 15057 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d8ca1b5399e14ad98016d79abbc191df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:28.896175 15055 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36756 (request call id 33) took 1921 ms. Trace:
I20240501 19:57:28.896238 15055 rpcz_store.cc:276] 0501 19:57:26.974264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.207079 (+1232815us) service_pool.cc:225] Handling call
0501 19:57:28.346671 (+139592us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.896154 (+549483us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":208232,"Hive Metastore.run_cpu_time_us":136,"Hive Metastore.run_wall_time_us":48828}
I20240501 19:57:28.896809 15062 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36756 (request call id 35) took 1867 ms. Trace:
I20240501 19:57:28.896883 15062 rpcz_store.cc:276] 0501 19:57:27.029338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:28.230337 (+1200999us) service_pool.cc:225] Handling call
0501 19:57:28.348254 (+117917us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:28.896798 (+548544us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":255518,"Hive Metastore.run_cpu_time_us":136,"Hive Metastore.run_wall_time_us":46303}
I20240501 19:57:28.899647 15055 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.899698 15060 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_abc8d72a6cc74062a62861c18c185a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.899794 15306 tablet_service.cc:1467] Processing CreateTablet for tablet f8c7c7763c08463ba9d1056d39988b4d (DEFAULT_TABLE table=default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]), partition=RANGE (key) PARTITION UNBOUNDED
W20240501 19:57:28.900002 15055 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 consensus state has no leader
I20240501 19:57:28.900086 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c7c7763c08463ba9d1056d39988b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.900204 15055 master_service.cc:946] ReplaceTablet: received request to replace tablet 4814dbbccd08455aa0b7381de1a34616 from {username='slave'} at 127.0.0.1:36756
I20240501 19:57:28.900563 15055 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_206463f213f945b2a74cbbcddbada5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.901115 15062 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:28.901245 15062 catalog_manager.cc:5484] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:28.902541 15578 tablet_service.cc:1467] Processing CreateTablet for tablet f8c7c7763c08463ba9d1056d39988b4d (DEFAULT_TABLE table=default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.902817 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c7c7763c08463ba9d1056d39988b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.902828 15444 tablet_service.cc:1467] Processing CreateTablet for tablet f8c7c7763c08463ba9d1056d39988b4d (DEFAULT_TABLE table=default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.903106 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c7c7763c08463ba9d1056d39988b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.905264 15906 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.906297 15906 tablet_bootstrap.cc:654] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.906318 15910 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.907315 15910 tablet_bootstrap.cc:654] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.908215 15061 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 539033016933451fa52037d581015148
I20240501 19:57:28.908553 15061 catalog_manager.cc:6554] ReplaceTablet: tablet 539033016933451fa52037d581015148 deleted and replaced by tablet 8f66c44e03c141a3a09fde32271b8829
I20240501 19:57:28.908746 15061 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0ef1c033257e4e358ccbe701cce981e2" } modify_external_catalogs: true
I20240501 19:57:28.908915 15061 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36756:
table { table_name: "default.Table_0ef1c033257e4e358ccbe701cce981e2" } modify_external_catalogs: true
I20240501 19:57:28.909077 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 539033016933451fa52037d581015148 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f66c44e03c141a3a09fde32271b8829) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.909261 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 539033016933451fa52037d581015148 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f66c44e03c141a3a09fde32271b8829) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.909435 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:40964
I20240501 19:57:28.909598 15936 tablet_replica.cc:330] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.909732 15936 raft_consensus.cc:2229] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.909844 15936 raft_consensus.cc:2258] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.910028 15934 tablet_replica.cc:330] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.910183 15934 raft_consensus.cc:2229] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:28.910284 15984 consensus_queue.cc:1035] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.910395 15934 raft_consensus.cc:2258] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.910490 15936 ts_tablet_manager.cc:1875] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.910909 15934 ts_tablet_manager.cc:1875] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.911701 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 539033016933451fa52037d581015148 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f66c44e03c141a3a09fde32271b8829) from {username='slave'} at 127.0.0.1:56688
I20240501 19:57:28.912925 15936 ts_tablet_manager.cc:1888] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.913003 15936 log.cc:1196] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/539033016933451fa52037d581015148
I20240501 19:57:28.913257 15936 ts_tablet_manager.cc:1909] T 539033016933451fa52037d581015148 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.913262 15934 ts_tablet_manager.cc:1888] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.913324 15934 log.cc:1196] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/539033016933451fa52037d581015148
I20240501 19:57:28.913569 15934 ts_tablet_manager.cc:1909] T 539033016933451fa52037d581015148 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.914180 15062 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36756:
name: "default.Table_7dee854303424e5db523b5680176f6c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:28.914423 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 9384878f4f834c9d90db68234d8755ae with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:49442
I20240501 19:57:28.915849 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 539033016933451fa52037d581015148 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:28.916078 15934 tablet_replica.cc:330] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:28.916224 15934 raft_consensus.cc:2229] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.916328 15934 raft_consensus.cc:2258] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.916819 15934 ts_tablet_manager.cc:1875] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.919093 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66c44e03c141a3a09fde32271b8829 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.919230 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66c44e03c141a3a09fde32271b8829 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.919428 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66c44e03c141a3a09fde32271b8829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.919389 15984 consensus_queue.cc:1035] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:28.919495 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66c44e03c141a3a09fde32271b8829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.921483 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 8f66c44e03c141a3a09fde32271b8829 (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:28.921747 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f66c44e03c141a3a09fde32271b8829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:28.922398 15936 tablet_replica.cc:330] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:28.922523 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 539033016933451fa52037d581015148 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:28.922659 15936 raft_consensus.cc:2229] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.922821 15936 raft_consensus.cc:2258] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.923529 15910 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.923641 15910 ts_tablet_manager.cc:1367] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:57:28.924132 15910 raft_consensus.cc:348] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.924240 15910 raft_consensus.cc:374] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.924273 15910 raft_consensus.cc:729] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.924391 15910 consensus_queue.cc:260] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.924643 15910 ts_tablet_manager.cc:1398] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.924814 15910 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:28.925824 15910 tablet_bootstrap.cc:654] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.926054 15936 ts_tablet_manager.cc:1875] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.928050 15935 tablet_replica.cc:330] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.928172 15935 raft_consensus.cc:2229] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.928265 15935 raft_consensus.cc:2258] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.928395 15936 ts_tablet_manager.cc:1888] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:28.928467 15936 log.cc:1196] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/9384878f4f834c9d90db68234d8755ae
I20240501 19:57:28.928681 15936 ts_tablet_manager.cc:1909] T 9384878f4f834c9d90db68234d8755ae P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:28.929343 15935 ts_tablet_manager.cc:1875] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.931501 15935 ts_tablet_manager.cc:1888] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:28.932617 15935 log.cc:1196] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/9384878f4f834c9d90db68234d8755ae
I20240501 19:57:28.932734 15051 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]) successfully deleted
I20240501 19:57:28.932998 15935 ts_tablet_manager.cc:1909] T 9384878f4f834c9d90db68234d8755ae P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.933738 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]) successfully deleted
I20240501 19:57:28.934041 15935 tablet_replica.cc:330] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:28.934118 15914 tablet_bootstrap.cc:492] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.934232 15914 ts_tablet_manager.cc:1367] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.000s
I20240501 19:57:28.934410 15935 raft_consensus.cc:2229] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:28.934473 15934 ts_tablet_manager.cc:1888] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:28.934564 15934 log.cc:1196] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/9384878f4f834c9d90db68234d8755ae
I20240501 19:57:28.934671 15935 raft_consensus.cc:2258] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:28.934791 15934 ts_tablet_manager.cc:1909] T 9384878f4f834c9d90db68234d8755ae P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:28.934773 15914 raft_consensus.cc:348] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.934897 15914 raft_consensus.cc:374] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.934935 15914 raft_consensus.cc:729] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.935070 15914 consensus_queue.cc:260] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.935395 15914 ts_tablet_manager.cc:1398] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.935459 15048 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad]) successfully deleted
I20240501 19:57:28.935752 15910 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:28.935828 15910 ts_tablet_manager.cc:1367] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:28.935971 15906 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.936055 15906 ts_tablet_manager.cc:1367] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.002s
I20240501 19:57:28.936250 15910 raft_consensus.cc:348] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.936363 15910 raft_consensus.cc:374] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.936398 15910 raft_consensus.cc:729] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.936514 15910 consensus_queue.cc:260] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.936555 15906 raft_consensus.cc:348] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.936664 15906 raft_consensus.cc:374] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.936691 15914 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.936694 15906 raft_consensus.cc:729] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.936786 15910 ts_tablet_manager.cc:1398] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.936815 15906 consensus_queue.cc:260] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.937112 15906 ts_tablet_manager.cc:1398] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.937317 15906 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:28.938244 15906 tablet_bootstrap.cc:654] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.940570 15914 tablet_bootstrap.cc:654] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.941740 15914 tablet_bootstrap.cc:492] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.941938 15914 ts_tablet_manager.cc:1367] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:28.942593 15914 raft_consensus.cc:348] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.942759 15914 raft_consensus.cc:374] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.942884 15914 raft_consensus.cc:729] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.943071 15914 consensus_queue.cc:260] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:28.943444 15914 ts_tablet_manager.cc:1398] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.943651 15914 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:28.944622 15914 tablet_bootstrap.cc:654] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:28.945379 15906 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:28.945487 15906 ts_tablet_manager.cc:1367] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 19:57:28.945926 15906 raft_consensus.cc:348] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.946038 15906 raft_consensus.cc:374] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.946074 15906 raft_consensus.cc:729] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.946219 15906 consensus_queue.cc:260] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.946518 15906 ts_tablet_manager.cc:1398] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:28.948680 15935 ts_tablet_manager.cc:1875] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:28.951014 15935 ts_tablet_manager.cc:1888] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:28.951097 15935 log.cc:1196] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/539033016933451fa52037d581015148
I20240501 19:57:28.951334 15935 ts_tablet_manager.cc:1909] T 539033016933451fa52037d581015148 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:28.951887 15914 tablet_bootstrap.cc:492] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:28.951987 15914 ts_tablet_manager.cc:1367] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:28.952455 15914 raft_consensus.cc:348] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.952574 15914 raft_consensus.cc:374] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:28.952610 15914 raft_consensus.cc:729] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:28.952746 15914 consensus_queue.cc:260] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:28.953043 15914 ts_tablet_manager.cc:1398] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:28.963611 15050 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 539033016933451fa52037d581015148 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:28.997687 15059 catalog_manager.cc:3408] renamed table default.table_0441f7d080e84fb792251357a57aa031 [id=4b0264d091414eac90e34e72e5ced992] in HMS: new name default.table_43971f18a04d42be9b5f69e32a2031ed
I20240501 19:57:29.001641 15968 consensus_queue.cc:1035] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.009145 15741 consensus_queue.cc:1035] T 71c0ca8638f14b89a2041c10a2e2206d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.017253 15752 raft_consensus.cc:482] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:29.017391 15752 raft_consensus.cc:504] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.017825 15752 leader_election.cc:288] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.018363 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a049fb6ddfd4cf49571cfbe9f4375a1" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:29.018536 15325 raft_consensus.cc:2454] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.018883 15533 leader_election.cc:302] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.019457 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a049fb6ddfd4cf49571cfbe9f4375a1" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:29.019616 15463 raft_consensus.cc:2454] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.020076 15752 raft_consensus.cc:2781] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:29.020145 15752 raft_consensus.cc:482] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:29.020187 15752 raft_consensus.cc:3037] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.021409 15752 raft_consensus.cc:504] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.021796 15752 leader_election.cc:288] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.022262 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a049fb6ddfd4cf49571cfbe9f4375a1" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:29.022389 15463 raft_consensus.cc:3037] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.022958 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a049fb6ddfd4cf49571cfbe9f4375a1" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:29.023090 15325 raft_consensus.cc:3037] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.023494 15463 raft_consensus.cc:2454] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.023927 15536 leader_election.cc:302] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.024039 15325 raft_consensus.cc:2454] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.024098 15752 raft_consensus.cc:2781] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:29.024170 15752 raft_consensus.cc:686] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:29.024302 15752 consensus_queue.cc:237] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.056430 15064 catalog_manager.cc:3408] renamed table default.table_0c593340e8374d2d809dda0943eb4dd7 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2] in HMS: new name default.table_5481ef3ce5fa4ea2b569d6ac2a569b75
I20240501 19:57:29.073081 15723 raft_consensus.cc:482] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:29.073254 15723 raft_consensus.cc:504] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.073784 15723 leader_election.cc:288] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.074256 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c7c7763c08463ba9d1056d39988b4d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:29.074447 15325 raft_consensus.cc:2454] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.075085 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c7c7763c08463ba9d1056d39988b4d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:29.075268 15463 raft_consensus.cc:2454] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.075850 15533 leader_election.cc:302] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.076046 15723 raft_consensus.cc:2781] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:29.076097 15723 raft_consensus.cc:482] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:29.076134 15723 raft_consensus.cc:3037] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.087783 15741 raft_consensus.cc:482] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:29.087951 15741 raft_consensus.cc:504] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.088443 15741 leader_election.cc:288] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:29.089550 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66c44e03c141a3a09fde32271b8829" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:29.089702 15325 raft_consensus.cc:2454] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:29.089942 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66c44e03c141a3a09fde32271b8829" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:29.090080 15598 raft_consensus.cc:2454] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:29.090476 15399 leader_election.cc:302] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.090801 15741 raft_consensus.cc:2781] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:29.090859 15741 raft_consensus.cc:482] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:29.090899 15741 raft_consensus.cc:3037] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.093022 15968 consensus_queue.cc:1035] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.106102 15723 raft_consensus.cc:504] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.106559 15723 leader_election.cc:288] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.107250 15741 raft_consensus.cc:504] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.107614 15741 leader_election.cc:288] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:29.108474 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66c44e03c141a3a09fde32271b8829" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:29.108594 15598 raft_consensus.cc:3037] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.109701 15598 raft_consensus.cc:2454] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:29.110018 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c7c7763c08463ba9d1056d39988b4d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:29.110137 15325 raft_consensus.cc:3037] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.110818 15399 leader_election.cc:302] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.110996 15968 consensus_queue.cc:1035] T 2ed5f03a8ff34913a59c475bd7ec83cf P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.111264 15968 raft_consensus.cc:2781] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:29.111342 15968 raft_consensus.cc:686] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:29.111486 15968 consensus_queue.cc:237] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.111605 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f66c44e03c141a3a09fde32271b8829" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:29.111737 15326 raft_consensus.cc:3037] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.111894 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c7c7763c08463ba9d1056d39988b4d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:29.112010 15463 raft_consensus.cc:3037] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.119415 15723 consensus_queue.cc:1035] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.124573 15752 consensus_queue.cc:1035] T aa19839336314324b4160d1f3c80a92d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.148402 15865 consensus_queue.cc:1035] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.152910 15984 consensus_queue.cc:1035] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.179517 15326 raft_consensus.cc:2454] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:29.179509 15463 raft_consensus.cc:2454] T f8c7c7763c08463ba9d1056d39988b4d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.179584 15325 raft_consensus.cc:2454] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.180040 15533 leader_election.cc:302] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.181308 16021 raft_consensus.cc:2781] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:29.181478 16021 raft_consensus.cc:686] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:29.181701 16021 consensus_queue.cc:237] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.184619 15056 catalog_manager.cc:3408] renamed table default.table_ea70d92de3fc40b4b3e8c54d6e5b6899 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42] in HMS: new name default.table_9254e341432a42c29665f212b2913ff6
I20240501 19:57:29.196714 16021 raft_consensus.cc:482] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:29.196857 16021 raft_consensus.cc:504] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.197261 16021 leader_election.cc:288] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.197638 15920 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240501 19:57:29.197692 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f6808094c74a4e8b3b7d092d863042" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:29.197794 15326 raft_consensus.cc:2454] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.198067 15533 leader_election.cc:302] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.198488 16021 raft_consensus.cc:2781] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:29.198545 16021 raft_consensus.cc:482] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:29.198590 16021 raft_consensus.cc:3037] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.199050 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f6808094c74a4e8b3b7d092d863042" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:29.199115 15932 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240501 19:57:29.199193 15463 raft_consensus.cc:2454] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:29.200501 15918 tablet.cc:1722] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240501 19:57:29.207520 16026 tablet.cc:1722] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.210173 16030 tablet.cc:1722] T 9a6cb71ea54e4cc2985b9d607109aba5 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.211040 16027 tablet.cc:1722] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.211326 16031 tablet.cc:1722] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.211750 16029 tablet.cc:1722] T 9a6cb71ea54e4cc2985b9d607109aba5 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.212218 16032 tablet.cc:1722] T 9a6cb71ea54e4cc2985b9d607109aba5 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:29.280664 16021 raft_consensus.cc:504] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.281250 16021 leader_election.cc:288] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:29.284904 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f6808094c74a4e8b3b7d092d863042" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:29.284899 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f6808094c74a4e8b3b7d092d863042" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:29.285043 15326 raft_consensus.cc:3037] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.285060 15462 raft_consensus.cc:3037] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.352893 15868 consensus_queue.cc:1035] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.356436 15868 consensus_queue.cc:1035] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.374055 15462 raft_consensus.cc:2454] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.377112 15536 leader_election.cc:302] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:29.377357 16021 raft_consensus.cc:2781] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:29.377460 16021 raft_consensus.cc:686] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:29.377599 16021 consensus_queue.cc:237] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:29.378742 15741 consensus_queue.cc:1035] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.380782 15326 raft_consensus.cc:2454] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:29.385908 15968 consensus_queue.cc:1035] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.427647 14720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu with pid 15032
W20240501 19:57:29.438689 15653 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.61:46321)
W20240501 19:57:29.439013 15653 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.61:46321)
W20240501 19:57:29.439268 15653 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.61:46321)
W20240501 19:57:29.440085 15653 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.61:46321)
I20240501 19:57:29.438519 15720 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: recv got EOF from 127.14.96.61:46321 (error 108),
I20240501 19:57:29.440831 15720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:29.528995 16021 consensus_queue.cc:1035] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.533083 15723 consensus_queue.cc:1035] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.556706 16017 consensus_queue.cc:1035] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.561324 15741 consensus_queue.cc:1035] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.631870 15723 consensus_queue.cc:1035] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.635655 15752 consensus_queue.cc:1035] T f8c7c7763c08463ba9d1056d39988b4d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.707300 16050 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:29.707518 16050 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:29.708577 16050 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:29.710724 16053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:29.710914 16053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:29.712652 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 2 candidate_status { last_received { term: 1 index: 100 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:29.712787 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 1.
I20240501 19:57:29.713177 15789 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:29.713843 16053 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:29.713932 16053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:57:29.713980 16053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:29.714015 16053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:29.716370 15787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:29.717265 15787 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:29.718348 14977 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:29.720108 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 100 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:29.768159 16053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:29.768884 16053 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:29.769351 15807 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9f177158f524f0183f6eec721ea83e4 in current term 2: Already voted for candidate 62106a6dbb3846df9d3b29ab99fb3763 in this term.
I20240501 19:57:29.769714 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 2 candidate_status { last_received { term: 1 index: 100 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:29.769845 15003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:29.770562 14978 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:29.771191 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 2.
I20240501 19:57:29.771389 16050 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:57:29.771664 15789 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:29.772004 16053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:29.772104 16053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 LEADER]: Becoming Leader. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Running, Role: LEADER
I20240501 19:57:29.772440 16053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
W20240501 19:57:29.773340 15787 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:29.773869 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:29.773998 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:29.774344 16058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:29.776569 16058 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:29.777187 16058 catalog_manager.cc:669] Loaded metadata for table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]
I20240501 19:57:29.777369 15752 consensus_queue.cc:1035] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.777556 16058 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:29.777822 16058 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:29.778018 16058 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:29.778254 16058 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:29.778507 16058 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:29.778725 16058 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:29.778951 16058 catalog_manager.cc:669] Loaded metadata for table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]
I20240501 19:57:29.779167 16058 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:29.779515 16058 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:29.779736 16058 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:29.779958 16058 catalog_manager.cc:669] Loaded metadata for table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]
I20240501 19:57:29.780185 16058 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:29.780444 16058 catalog_manager.cc:669] Loaded metadata for table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]
I20240501 19:57:29.782424 15752 consensus_queue.cc:1035] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:29.782909 16058 tablet_loader.cc:96] loaded metadata for tablet 0f838413cd7a4975918e88551c465f18 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:29.783139 16058 tablet_loader.cc:96] loaded metadata for tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60])
I20240501 19:57:29.783483 16058 tablet_loader.cc:96] loaded metadata for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:29.783687 16058 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:29.783874 16058 tablet_loader.cc:96] loaded metadata for tablet 34f6808094c74a4e8b3b7d092d863042 (table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96])
I20240501 19:57:29.784288 16058 tablet_loader.cc:96] loaded metadata for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:29.784626 16058 tablet_loader.cc:96] loaded metadata for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33])
I20240501 19:57:29.784798 16058 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:29.785007 16058 tablet_loader.cc:96] loaded metadata for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:29.785203 16058 tablet_loader.cc:96] loaded metadata for tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:29.785553 16058 tablet_loader.cc:96] loaded metadata for tablet 8f66c44e03c141a3a09fde32271b8829 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f])
I20240501 19:57:29.785764 16058 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:29.785943 16058 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:29.786398 16058 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:29.786861 16058 tablet_loader.cc:96] loaded metadata for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:29.787384 16058 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:29.787544 16058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:29.788331 16058 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:29.788376 16058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:29.789299 16058 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:29.789973 16058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Loaded TSK: 0
I20240501 19:57:29.790789 16058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:29.791473 16058 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:29.792356 16058 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 33
I20240501 19:57:29.793954 15796 master_service.cc:946] ReplaceTablet: received request to replace tablet 78becd64ac1d4da5b4ad1474e5f41c5e from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:29.794087 15797 master_service.cc:946] ReplaceTablet: received request to replace tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:29.794422 15802 master_service.cc:946] ReplaceTablet: received request to replace tablet 8f66c44e03c141a3a09fde32271b8829 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:29.795363 15798 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_758f18364c874f0a824123020aad0241" } modify_external_catalogs: true
I20240501 19:57:29.795550 15798 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_758f18364c874f0a824123020aad0241" } modify_external_catalogs: true
I20240501 19:57:29.795815 15799 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_a63257f60f4e4b9ca1d285937e0fe0e3" } modify_external_catalogs: true
I20240501 19:57:29.795914 15799 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_a63257f60f4e4b9ca1d285937e0fe0e3" } modify_external_catalogs: true
I20240501 19:57:29.796478 15803 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f838413cd7a4975918e88551c465f18 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:29.796415 15805 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_a283fbed2b5748f1b138b73c95bb2dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.796415 15804 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_7a325fdcd5224571b886231cf98de803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.797180 15801 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_3df0bbd8beda416987f50555069e12d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.797727 15800 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_83b626e1fcb24923aeb39c18f422d4dc" } modify_external_catalogs: true
I20240501 19:57:29.797838 15800 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_83b626e1fcb24923aeb39c18f422d4dc" } modify_external_catalogs: true
I20240501 19:57:29.798413 15003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Refusing update from remote peer 62106a6dbb3846df9d3b29ab99fb3763: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 2 index: 102. (index mismatch)
I20240501 19:57:29.799166 16053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
W20240501 19:57:29.799311 15787 consensus_peers.cc:485] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 -> Peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Couldn't send request to peer 72d8e36923de4d1e8a6de93e73c84658. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:29.801131 16050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:29.801263 16050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:29.802681 16061 mvcc.cc:205] Tried to move back new op lower bound from 7022974770369368064 to 7022974770271096832. Current Snapshot: MvccSnapshot[applied={T|T < 7022974770364444672}]
I20240501 19:57:29.803123 16055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:29.803221 16055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:29.803879 15796 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 78becd64ac1d4da5b4ad1474e5f41c5e
I20240501 19:57:29.804682 15803 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 0f838413cd7a4975918e88551c465f18
I20240501 19:57:29.805058 15796 catalog_manager.cc:6554] ReplaceTablet: tablet 78becd64ac1d4da5b4ad1474e5f41c5e deleted and replaced by tablet fe8ac073d20b45138610f9f4ffad1138
I20240501 19:57:29.805142 15803 catalog_manager.cc:6554] ReplaceTablet: tablet 0f838413cd7a4975918e88551c465f18 deleted and replaced by tablet ec1ef46612cd4545a753bd4700897535
I20240501 19:57:29.806181 15797 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 2a049fb6ddfd4cf49571cfbe9f4375a1
I20240501 19:57:29.806514 15797 catalog_manager.cc:6554] ReplaceTablet: tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 deleted and replaced by tablet e896ddb02b824d6c9c5519c457f537bc
I20240501 19:57:29.806521 16050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:29.806640 16050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:29.806656 15797 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f838413cd7a4975918e88551c465f18 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:29.807116 15802 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 8f66c44e03c141a3a09fde32271b8829
I20240501 19:57:29.807390 15802 catalog_manager.cc:6554] ReplaceTablet: tablet 8f66c44e03c141a3a09fde32271b8829 deleted and replaced by tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:29.809274 15796 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_b16d0f8f6d3341c8bbd83fc19a8c3771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.809603 15803 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_fb234d1b9ddb449a96ad2affdb553574"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.811363 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 with delete_type TABLET_DATA_DELETED (replaced by tablet e896ddb02b824d6c9c5519c457f537bc) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:29.811470 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f838413cd7a4975918e88551c465f18 with delete_type TABLET_DATA_DELETED (replaced by tablet ec1ef46612cd4545a753bd4700897535) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:29.811595 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66c44e03c141a3a09fde32271b8829 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:29.811401 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e with delete_type TABLET_DATA_DELETED (replaced by tablet fe8ac073d20b45138610f9f4ffad1138) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:29.813377 16076 tablet_replica.cc:330] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:29.813548 16076 raft_consensus.cc:2229] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:29.813665 16076 raft_consensus.cc:2258] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.813620 15802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_4907ae96e84044d5ace4ea027902eecb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:29.813669 15797 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_8060d3196acf479897cfe5f6f547dba9" } modify_external_catalogs: true
I20240501 19:57:29.813791 15797 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_8060d3196acf479897cfe5f6f547dba9" } modify_external_catalogs: true
I20240501 19:57:29.814294 16076 ts_tablet_manager.cc:1875] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:29.815898 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 with delete_type TABLET_DATA_DELETED (replaced by tablet e896ddb02b824d6c9c5519c457f537bc) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:29.815951 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e with delete_type TABLET_DATA_DELETED (replaced by tablet fe8ac073d20b45138610f9f4ffad1138) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:29.816118 15577 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66c44e03c141a3a09fde32271b8829 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:29.815922 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f838413cd7a4975918e88551c465f18 with delete_type TABLET_DATA_DELETED (replaced by tablet ec1ef46612cd4545a753bd4700897535) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:29.817004 16077 tablet_replica.cc:330] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:29.817162 16077 raft_consensus.cc:2229] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:29.817365 16077 raft_consensus.cc:2258] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.817852 16077 ts_tablet_manager.cc:1875] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:29.822052 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.822376 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f838413cd7a4975918e88551c465f18 with delete_type TABLET_DATA_DELETED (replaced by tablet ec1ef46612cd4545a753bd4700897535) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:29.822433 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f66c44e03c141a3a09fde32271b8829 with delete_type TABLET_DATA_DELETED (replaced by tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:29.822427 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.822665 15443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 with delete_type TABLET_DATA_DELETED (replaced by tablet e896ddb02b824d6c9c5519c457f537bc) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:29.822798 15440 tablet_service.cc:1467] Processing CreateTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (DEFAULT_TABLE table=default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.822978 15303 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef46612cd4545a753bd4700897535 (DEFAULT_TABLE table=default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.823096 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.823616 16079 tablet_replica.cc:330] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:29.824070 16079 raft_consensus.cc:2229] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:29.822468 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 78becd64ac1d4da5b4ad1474e5f41c5e with delete_type TABLET_DATA_DELETED (replaced by tablet fe8ac073d20b45138610f9f4ffad1138) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:29.825103 16079 raft_consensus.cc:2258] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.825946 16079 ts_tablet_manager.cc:1875] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:29.826222 15577 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef46612cd4545a753bd4700897535 (DEFAULT_TABLE table=default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.826925 15444 tablet_service.cc:1467] Processing CreateTablet for tablet fe8ac073d20b45138610f9f4ffad1138 (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.827210 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8ac073d20b45138610f9f4ffad1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.827910 15304 tablet_service.cc:1467] Processing CreateTablet for tablet fe8ac073d20b45138610f9f4ffad1138 (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.828330 15443 tablet_service.cc:1467] Processing CreateTablet for tablet e896ddb02b824d6c9c5519c457f537bc (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.828466 15440 tablet_service.cc:1467] Processing CreateTablet for tablet ec1ef46612cd4545a753bd4700897535 (DEFAULT_TABLE table=default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.828563 15579 tablet_service.cc:1467] Processing CreateTablet for tablet fe8ac073d20b45138610f9f4ffad1138 (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.828676 15580 tablet_service.cc:1467] Processing CreateTablet for tablet e896ddb02b824d6c9c5519c457f537bc (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.828783 16080 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:29.827924 15305 tablet_service.cc:1467] Processing CreateTablet for tablet e896ddb02b824d6c9c5519c457f537bc (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:29.829842 16080 tablet_bootstrap.cc:654] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.832039 16080 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:29.832142 16080 ts_tablet_manager.cc:1367] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:29.832679 16080 raft_consensus.cc:348] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.832929 16080 raft_consensus.cc:374] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.832996 16080 raft_consensus.cc:729] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.833180 16080 consensus_queue.cc:260] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.833623 16080 ts_tablet_manager.cc:1398] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.874567 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.874926 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef46612cd4545a753bd4700897535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.875129 15304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8ac073d20b45138610f9f4ffad1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.875353 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e896ddb02b824d6c9c5519c457f537bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.877632 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef46612cd4545a753bd4700897535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.877635 16076 ts_tablet_manager.cc:1888] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:29.877729 16076 log.cc:1196] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/2a049fb6ddfd4cf49571cfbe9f4375a1
I20240501 19:57:29.878036 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8ac073d20b45138610f9f4ffad1138. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.878048 16076 ts_tablet_manager.cc:1909] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:29.878371 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e896ddb02b824d6c9c5519c457f537bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.878938 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.879334 16082 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:29.880267 16082 tablet_bootstrap.cc:654] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.887552 16082 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:29.887671 16082 ts_tablet_manager.cc:1367] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:29.888217 16082 raft_consensus.cc:348] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.888343 16082 raft_consensus.cc:374] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.888370 16082 raft_consensus.cc:729] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.888532 16082 consensus_queue.cc:260] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.888829 16082 ts_tablet_manager.cc:1398] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.890448 16082 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:29.892441 16082 tablet_bootstrap.cc:654] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.893909 16082 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:29.894001 16082 ts_tablet_manager.cc:1367] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:29.894579 16082 raft_consensus.cc:348] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.894692 16082 raft_consensus.cc:374] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.894757 16082 raft_consensus.cc:729] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.894901 16082 consensus_queue.cc:260] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.895254 16082 ts_tablet_manager.cc:1398] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.895488 16082 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:29.898511 16082 tablet_bootstrap.cc:654] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.899619 16082 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:29.899706 16082 ts_tablet_manager.cc:1367] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:29.902029 15741 raft_consensus.cc:482] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:29.902130 15741 raft_consensus.cc:504] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.902509 15741 leader_election.cc:288] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:29.903120 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b9f8b8fe34fe49c6cc0cf0ca2e07d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:29.903237 15326 raft_consensus.cc:2454] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
W20240501 19:57:29.903254 15399 leader_election.cc:341] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Not found: Tablet not found: 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:29.903576 15397 leader_election.cc:302] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters: 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:29.903813 15741 raft_consensus.cc:2781] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:29.903868 15741 raft_consensus.cc:482] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:29.903904 15741 raft_consensus.cc:3037] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.900756 16082 raft_consensus.cc:348] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.904982 16082 raft_consensus.cc:374] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.905036 16082 raft_consensus.cc:729] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.905177 16082 consensus_queue.cc:260] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.905462 16082 ts_tablet_manager.cc:1398] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:29.905697 16082 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:29.907979 16082 tablet_bootstrap.cc:654] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.909312 16082 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:29.909415 16082 ts_tablet_manager.cc:1367] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:29.909878 16082 raft_consensus.cc:348] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.909988 16082 raft_consensus.cc:374] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.910023 16082 raft_consensus.cc:729] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.910128 16082 consensus_queue.cc:260] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.910379 16082 ts_tablet_manager.cc:1398] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.968771 16076 tablet_replica.cc:330] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:29.969018 16076 raft_consensus.cc:2229] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:29.969056 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:29.969292 16076 raft_consensus.cc:2258] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.969785 16076 ts_tablet_manager.cc:1875] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:29.974083 15741 raft_consensus.cc:504] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.974247 16077 ts_tablet_manager.cc:1888] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:29.974349 16077 log.cc:1196] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/2a049fb6ddfd4cf49571cfbe9f4375a1
I20240501 19:57:29.974618 16077 ts_tablet_manager.cc:1909] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:29.974603 15741 leader_election.cc:288] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:29.975143 16079 ts_tablet_manager.cc:1888] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:29.975217 16079 log.cc:1196] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/0f838413cd7a4975918e88551c465f18
I20240501 19:57:29.975483 16079 ts_tablet_manager.cc:1909] T 0f838413cd7a4975918e88551c465f18 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:29.975857 15443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e896ddb02b824d6c9c5519c457f537bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.978710 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1ef46612cd4545a753bd4700897535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.979709 16088 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:29.980615 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b9f8b8fe34fe49c6cc0cf0ca2e07d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:29.981523 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b9f8b8fe34fe49c6cc0cf0ca2e07d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:29.981654 15326 raft_consensus.cc:3037] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.982069 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:29.980881 16088 tablet_bootstrap.cc:654] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.982380 16080 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:29.981071 16077 tablet_replica.cc:330] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
W20240501 19:57:29.983040 15399 leader_election.cc:341] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:29.983259 16077 raft_consensus.cc:2229] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:29.983386 16080 tablet_bootstrap.cc:654] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.983497 16077 raft_consensus.cc:2258] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.983911 16077 ts_tablet_manager.cc:1875] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:29.984359 16080 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:29.984364 16088 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:29.984442 16088 ts_tablet_manager.cc:1367] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:29.984453 16080 ts_tablet_manager.cc:1367] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:29.984899 16088 raft_consensus.cc:348] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.984897 16080 raft_consensus.cc:348] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.984995 16088 raft_consensus.cc:374] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.984995 16080 raft_consensus.cc:374] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.985028 16080 raft_consensus.cc:729] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.985028 16088 raft_consensus.cc:729] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.985146 16080 consensus_queue.cc:260] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.985461 16080 ts_tablet_manager.cc:1398] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.985272 16088 consensus_queue.cc:260] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.985684 16080 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:29.985843 16088 ts_tablet_manager.cc:1398] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.986011 16088 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:29.986584 16080 tablet_bootstrap.cc:654] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.987383 16088 tablet_bootstrap.cc:654] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.989508 16080 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:29.989599 16080 ts_tablet_manager.cc:1367] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:29.990056 16080 raft_consensus.cc:348] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.990166 16080 raft_consensus.cc:374] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.990219 16080 raft_consensus.cc:729] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.990347 16080 consensus_queue.cc:260] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.990661 16080 ts_tablet_manager.cc:1398] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.990809 16080 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:29.991731 16080 tablet_bootstrap.cc:654] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.993185 16080 tablet_bootstrap.cc:492] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:29.993234 16088 tablet_bootstrap.cc:492] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:29.993326 16088 ts_tablet_manager.cc:1367] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:29.993353 16080 ts_tablet_manager.cc:1367] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:29.993907 16080 raft_consensus.cc:348] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.994014 16080 raft_consensus.cc:374] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.994050 16080 raft_consensus.cc:729] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.994158 16088 raft_consensus.cc:348] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.994271 16088 raft_consensus.cc:374] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.994226 16080 consensus_queue.cc:260] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.994347 16088 raft_consensus.cc:729] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.994539 16080 ts_tablet_manager.cc:1398] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:29.994520 16088 consensus_queue.cc:260] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.994876 16088 ts_tablet_manager.cc:1398] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.995090 16088 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:29.996217 16088 tablet_bootstrap.cc:654] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.997709 16088 tablet_bootstrap.cc:492] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:29.997838 16088 ts_tablet_manager.cc:1367] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:29.998322 16088 raft_consensus.cc:348] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.998422 16088 raft_consensus.cc:374] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.998489 16088 raft_consensus.cc:729] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.998625 16088 consensus_queue.cc:260] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:29.999006 16088 ts_tablet_manager.cc:1398] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.999217 16088 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:30.000151 16088 tablet_bootstrap.cc:654] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.000232 16076 ts_tablet_manager.cc:1888] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:30.000353 16076 log.cc:1196] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/0f838413cd7a4975918e88551c465f18
I20240501 19:57:30.000669 16076 ts_tablet_manager.cc:1909] T 0f838413cd7a4975918e88551c465f18 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:30.001663 16088 tablet_bootstrap.cc:492] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:30.001750 16088 ts_tablet_manager.cc:1367] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.002142 16088 raft_consensus.cc:348] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.002244 16088 raft_consensus.cc:374] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.002316 16088 raft_consensus.cc:729] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.002490 16088 consensus_queue.cc:260] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.002800 16088 ts_tablet_manager.cc:1398] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.003973 16079 tablet_replica.cc:330] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:30.004129 16079 raft_consensus.cc:2229] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:30.004115 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 0f838413cd7a4975918e88551c465f18 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]) successfully deleted
I20240501 19:57:30.004365 16079 raft_consensus.cc:2258] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.005167 16079 ts_tablet_manager.cc:1875] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.024057 15752 raft_consensus.cc:482] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.024223 15752 raft_consensus.cc:504] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.024776 15752 leader_election.cc:288] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.025166 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef46612cd4545a753bd4700897535" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:30.025341 15325 raft_consensus.cc:2454] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:30.025678 15533 leader_election.cc:302] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.025887 15723 raft_consensus.cc:2781] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.025943 15723 raft_consensus.cc:482] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.025998 15723 raft_consensus.cc:3037] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.029349 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef46612cd4545a753bd4700897535" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:30.029537 15462 raft_consensus.cc:2454] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:30.073000 15326 raft_consensus.cc:2454] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:30.073004 15723 raft_consensus.cc:504] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.073576 15723 leader_election.cc:288] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.075547 15798 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]
I20240501 19:57:30.075695 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef46612cd4545a753bd4700897535" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:30.075846 15326 raft_consensus.cc:3037] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.076874 15326 raft_consensus.cc:2454] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:30.077342 15533 leader_election.cc:302] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.077592 16076 tablet_replica.cc:330] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:30.077733 16076 raft_consensus.cc:2229] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.077845 16076 raft_consensus.cc:2258] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.077927 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1ef46612cd4545a753bd4700897535" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:30.078055 15462 raft_consensus.cc:3037] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.078339 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 0f838413cd7a4975918e88551c465f18 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]) successfully deleted
I20240501 19:57:30.078869 16077 ts_tablet_manager.cc:1888] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:30.078984 16077 log.cc:1196] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/78becd64ac1d4da5b4ad1474e5f41c5e
I20240501 19:57:30.079027 15462 raft_consensus.cc:2454] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:30.079241 15723 raft_consensus.cc:2781] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.079267 16077 ts_tablet_manager.cc:1909] T 78becd64ac1d4da5b4ad1474e5f41c5e P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:30.079322 15723 raft_consensus.cc:686] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:30.079463 15723 consensus_queue.cc:237] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.079905 16076 ts_tablet_manager.cc:1875] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.081198 15397 leader_election.cc:302] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters: 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:30.081444 15741 raft_consensus.cc:2781] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.081517 15741 raft_consensus.cc:686] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:30.081638 15741 consensus_queue.cc:237] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.091325 16086 raft_consensus.cc:482] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.091481 16086 raft_consensus.cc:504] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.091964 16086 leader_election.cc:288] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.092480 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e896ddb02b824d6c9c5519c457f537bc" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:30.092628 15598 raft_consensus.cc:2454] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:30.092758 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e896ddb02b824d6c9c5519c457f537bc" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:30.092916 15462 raft_consensus.cc:2454] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:30.093160 15261 leader_election.cc:302] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.093353 16086 raft_consensus.cc:2781] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.093408 16086 raft_consensus.cc:482] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.093472 16086 raft_consensus.cc:3037] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.103778 16017 raft_consensus.cc:482] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.103925 16017 raft_consensus.cc:504] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.104434 16017 leader_election.cc:288] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:30.105500 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8ac073d20b45138610f9f4ffad1138" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:30.105654 15598 raft_consensus.cc:2454] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:30.106047 15399 leader_election.cc:302] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.106309 16017 raft_consensus.cc:2781] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.106360 16017 raft_consensus.cc:482] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.106398 16017 raft_consensus.cc:3037] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.106371 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8ac073d20b45138610f9f4ffad1138" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:30.106547 15326 raft_consensus.cc:2454] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:30.125756 16079 ts_tablet_manager.cc:1888] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.125883 16079 log.cc:1196] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/8f66c44e03c141a3a09fde32271b8829
I20240501 19:57:30.126202 16079 ts_tablet_manager.cc:1909] T 8f66c44e03c141a3a09fde32271b8829 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:30.126350 16077 tablet_replica.cc:330] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:30.126495 16077 raft_consensus.cc:2229] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.126597 16077 raft_consensus.cc:2258] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.126744 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:30.127596 16077 ts_tablet_manager.cc:1875] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.149235 16076 ts_tablet_manager.cc:1888] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.149369 16076 log.cc:1196] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/8f66c44e03c141a3a09fde32271b8829
I20240501 19:57:30.149631 16076 ts_tablet_manager.cc:1909] T 8f66c44e03c141a3a09fde32271b8829 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:30.178190 16086 raft_consensus.cc:504] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.178153 16017 raft_consensus.cc:504] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.178705 16086 leader_election.cc:288] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.178726 16017 leader_election.cc:288] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:30.179566 16079 tablet_replica.cc:330] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:30.179685 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e896ddb02b824d6c9c5519c457f537bc" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:30.179738 16079 raft_consensus.cc:2229] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.179812 15598 raft_consensus.cc:3037] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.179850 16079 raft_consensus.cc:2258] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.180172 16076 tablet_replica.cc:330] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:30.180306 16076 raft_consensus.cc:2229] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.180428 16076 raft_consensus.cc:2258] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.180523 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 8f66c44e03c141a3a09fde32271b8829 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.180770 15598 raft_consensus.cc:2454] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:30.180868 16076 ts_tablet_manager.cc:1875] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.181099 16079 ts_tablet_manager.cc:1875] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.181531 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 8f66c44e03c141a3a09fde32271b8829 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.181923 15261 leader_election.cc:302] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.182767 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e896ddb02b824d6c9c5519c457f537bc" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:30.182885 15462 raft_consensus.cc:3037] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.183086 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8ac073d20b45138610f9f4ffad1138" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:30.183197 15326 raft_consensus.cc:3037] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.184001 16086 raft_consensus.cc:2781] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.184096 16086 raft_consensus.cc:686] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:30.183907 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8ac073d20b45138610f9f4ffad1138" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:30.184221 15598 raft_consensus.cc:3037] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.184233 16086 consensus_queue.cc:237] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.186029 16077 ts_tablet_manager.cc:1888] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.186138 16077 log.cc:1196] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/8f66c44e03c141a3a09fde32271b8829
I20240501 19:57:30.186419 16077 ts_tablet_manager.cc:1909] T 8f66c44e03c141a3a09fde32271b8829 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:30.217541 14962 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:30.219396 14962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:30.273813 15462 raft_consensus.cc:2454] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:30.273887 16076 ts_tablet_manager.cc:1888] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:30.273892 16079 ts_tablet_manager.cc:1888] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.273963 16076 log.cc:1196] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/78becd64ac1d4da5b4ad1474e5f41c5e
I20240501 19:57:30.273983 16079 log.cc:1196] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/2a049fb6ddfd4cf49571cfbe9f4375a1
I20240501 19:57:30.274152 15598 raft_consensus.cc:2454] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:30.274242 16076 ts_tablet_manager.cc:1909] T 78becd64ac1d4da5b4ad1474e5f41c5e P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:30.274266 16079 ts_tablet_manager.cc:1909] T 2a049fb6ddfd4cf49571cfbe9f4375a1 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:30.274549 15326 raft_consensus.cc:2454] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:30.274693 15399 leader_election.cc:302] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:30.275007 16077 tablet_replica.cc:330] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:30.275187 16077 raft_consensus.cc:2229] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.275310 16077 raft_consensus.cc:2258] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.275373 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 8f66c44e03c141a3a09fde32271b8829 (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.275429 16017 raft_consensus.cc:2781] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.275509 16017 raft_consensus.cc:686] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:30.275653 16017 consensus_queue.cc:237] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.276050 16077 ts_tablet_manager.cc:1875] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.279120 16079 tablet_replica.cc:330] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:30.279192 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 2a049fb6ddfd4cf49571cfbe9f4375a1 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:30.279223 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:30.279300 16079 raft_consensus.cc:2229] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.279423 16079 raft_consensus.cc:2258] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.279781 16079 ts_tablet_manager.cc:1875] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.338563 16077 ts_tablet_manager.cc:1888] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:30.338667 16077 log.cc:1196] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/0f838413cd7a4975918e88551c465f18
I20240501 19:57:30.338922 16077 ts_tablet_manager.cc:1909] T 0f838413cd7a4975918e88551c465f18 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:30.339751 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 0f838413cd7a4975918e88551c465f18 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]) successfully deleted
I20240501 19:57:30.367597 16079 ts_tablet_manager.cc:1888] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:30.367700 16079 log.cc:1196] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/78becd64ac1d4da5b4ad1474e5f41c5e
I20240501 19:57:30.367944 16079 ts_tablet_manager.cc:1909] T 78becd64ac1d4da5b4ad1474e5f41c5e P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:30.376770 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 78becd64ac1d4da5b4ad1474e5f41c5e (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]) successfully deleted
I20240501 19:57:30.436180 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--webserver_interface=127.14.96.61
--webserver_port=42603
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:30.469025 15799 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]
W20240501 19:57:30.472749 15801 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3df0bbd8beda416987f50555069e12d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:30.472901 15804 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7a325fdcd5224571b886231cf98de803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:30.472909 15803 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_fb234d1b9ddb449a96ad2affdb553574 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:30.473170 15802 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4907ae96e84044d5ace4ea027902eecb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:30.473174 15796 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:30.472749 15805 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a283fbed2b5748f1b138b73c95bb2dc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:30.477170 15859 catalog_manager.cc:2706] Deleting table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96] in response to Hive Metastore notification log event 35
I20240501 19:57:30.478591 15723 raft_consensus.cc:482] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.478704 15723 raft_consensus.cc:504] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:30.479167 15723 leader_election.cc:288] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.479794 15598 raft_consensus.cc:3037] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.481513 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b9f8b8fe34fe49c6cc0cf0ca2e07d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:30.482578 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d3b9f8b8fe34fe49c6cc0cf0ca2e07d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:30.482748 15326 raft_consensus.cc:2379] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7f011d422e164c949a9d54d846b7c432 in current term 1: Already voted for candidate 68539c0ecd3e44e080cb53987d4f4c23 in this term.
I20240501 19:57:30.484426 15533 leader_election.cc:302] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f011d422e164c949a9d54d846b7c432; no voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23
I20240501 19:57:30.494953 15859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:30.495806 15859 catalog_manager.cc:2706] Deleting table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f] in response to Hive Metastore notification log event 36
I20240501 19:57:30.495838 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:30.496120 16076 tablet_replica.cc:330] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:30.496273 16076 raft_consensus.cc:2229] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.496395 16076 raft_consensus.cc:2258] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.496954 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:30.497066 16076 ts_tablet_manager.cc:1875] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.497149 16077 tablet_replica.cc:330] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:30.497285 16077 raft_consensus.cc:2229] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:30.497395 15440 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:30.497530 16077 raft_consensus.cc:2258] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.497718 16079 tablet_replica.cc:330] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:30.497849 16079 raft_consensus.cc:2229] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.497959 16079 raft_consensus.cc:2258] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.498631 16079 ts_tablet_manager.cc:1875] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.500063 16077 ts_tablet_manager.cc:1875] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.502238 15859 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:30.502871 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:30.503996 15799 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:30.504168 15440 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:30.504382 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:56694
W20240501 19:57:30.504590 15799 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e896ddb02b824d6c9c5519c457f537bc consensus state has no leader
I20240501 19:57:30.504987 15798 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_7dee854303424e5db523b5680176f6c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:30.505307 15510 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
I20240501 19:57:30.505379 15799 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:30.505416 15372 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
W20240501 19:57:30.505738 15799 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ec1ef46612cd4545a753bd4700897535 consensus state has no leader
I20240501 19:57:30.505998 15799 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_0ef1c033257e4e358ccbe701cce981e2" } modify_external_catalogs: true
I20240501 19:57:30.506112 15799 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_0ef1c033257e4e358ccbe701cce981e2" } modify_external_catalogs: true
I20240501 19:57:30.506899 15646 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
W20240501 19:57:30.536777 15788 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e896ddb02b824d6c9c5519c457f537bc consensus state has no leader
I20240501 19:57:30.570947 15723 raft_consensus.cc:2726] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:30.571640 16017 consensus_queue.cc:1035] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.573457 16077 ts_tablet_manager.cc:1888] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.573555 16077 log.cc:1196] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:30.573678 16076 ts_tablet_manager.cc:1888] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.573765 16076 log.cc:1196] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:30.573843 16077 ts_tablet_manager.cc:1909] T 34f6808094c74a4e8b3b7d092d863042 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:30.574033 16076 ts_tablet_manager.cc:1909] T 34f6808094c74a4e8b3b7d092d863042 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:30.574559 16077 tablet_replica.cc:330] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:30.574702 16077 raft_consensus.cc:2229] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.574743 16076 tablet_replica.cc:330] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:30.574785 16077 pending_rounds.cc:62] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Trying to abort 1 pending ops.
I20240501 19:57:30.574841 16077 pending_rounds.cc:68] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7022974771535745024 op_type: NO_OP noop_request { }
I20240501 19:57:30.574874 16076 raft_consensus.cc:2229] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:30.574913 16077 raft_consensus.cc:2866] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240501 19:57:30.574963 16077 raft_consensus.cc:2258] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.574966 16076 raft_consensus.cc:2258] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.575376 16076 ts_tablet_manager.cc:1875] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.575425 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 34f6808094c74a4e8b3b7d092d863042 (table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]) successfully deleted
I20240501 19:57:30.575640 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 34f6808094c74a4e8b3b7d092d863042 (table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]) successfully deleted
W20240501 19:57:30.575978 16106 consensus_queue.cc:1159] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7f011d422e164c949a9d54d846b7c432" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240501 19:57:30.576462 16077 ts_tablet_manager.cc:1875] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:30.577940 16079 ts_tablet_manager.cc:1888] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.578023 16079 log.cc:1196] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/34f6808094c74a4e8b3b7d092d863042
W20240501 19:57:30.578248 15397 consensus_peers.cc:485] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 -> Peer 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): Couldn't send request to peer 0c8dfc11dc7542f583644b9ac747545d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:30.578294 16079 ts_tablet_manager.cc:1909] T 34f6808094c74a4e8b3b7d092d863042 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
W20240501 19:57:30.578812 15399 consensus_peers.cc:485] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 -> Peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Couldn't send request to peer 7f011d422e164c949a9d54d846b7c432. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:57:30.587528 16100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:30.587806 16100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:30.587893 16100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:30.591465 16100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:30.591542 16100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:30.591579 16100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:30.591598 16100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:30.595826 16100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.96.61
--webserver_port=42603
--never_fsync=true
--heap_profile_path=/tmp/kudu.16100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:30.596894 16100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:30.598232 15723 consensus_queue.cc:1035] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.598331 16100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:30.601701 16021 consensus_queue.cc:1035] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.605715 16079 tablet_replica.cc:330] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:30.605870 16079 raft_consensus.cc:2229] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:30.605857 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 34f6808094c74a4e8b3b7d092d863042 (table default.table_758f18364c874f0a824123020aad0241 [id=cb06dd07345741c8a5bd80df323d4f96]) successfully deleted
I20240501 19:57:30.606815 16079 raft_consensus.cc:2258] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:30.607434 16079 ts_tablet_manager.cc:1875] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 19:57:30.608157 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:30.608521 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:30.609050 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:30.610096 16100 server_base.cc:1034] running on GCE node
I20240501 19:57:30.610726 16100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:30.611815 16100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:30.612926 16100 hybrid_clock.cc:648] HybridClock initialized: now 1714593450612910 us; error 33 us; skew 500 ppm
I20240501 19:57:30.615089 16100 webserver.cc:460] Webserver started at http://127.14.96.61:42603/ using document root <none> and password file <none>
I20240501 19:57:30.615729 16100 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:30.615850 16100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:30.619659 16100 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:30.621852 16125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:30.622964 16100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:30.623112 16100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:30.623580 16100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:30.648156 16100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:30.648918 16100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:30.649081 16100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:30.656504 16100 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.61:46321
I20240501 19:57:30.656527 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.61:46321 every 8 connection(s)
I20240501 19:57:30.657634 16100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
I20240501 19:57:30.661883 16178 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:30.664988 16178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap starting.
I20240501 19:57:30.667490 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16100
I20240501 19:57:30.668130 16079 ts_tablet_manager.cc:1888] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.668130 16077 ts_tablet_manager.cc:1888] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:30.668180 16076 ts_tablet_manager.cc:1888] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240501 19:57:30.668220 16077 log.cc:1196] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:30.668220 16079 log.cc:1196] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:30.668275 16076 log.cc:1196] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:30.668503 16077 ts_tablet_manager.cc:1909] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:30.668503 16079 ts_tablet_manager.cc:1909] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:30.668520 16076 ts_tablet_manager.cc:1909] T 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:30.674257 15787 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.674821 15790 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.679026 16178 log.cc:824] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:30.679598 16017 consensus_queue.cc:1035] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.685611 15741 consensus_queue.cc:1035] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.689174 15374 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:30.697252 15648 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:30.698112 15512 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:30.705192 15789 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d (table default.table_a63257f60f4e4b9ca1d285937e0fe0e3 [id=eb4cd54084b447d89532e8f1f500ba7f]) successfully deleted
I20240501 19:57:30.735725 16178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=72 ignored=0} mutations{seen=83 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:30.736169 16178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap complete.
I20240501 19:57:30.739576 16178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:30.739953 16178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.740412 16178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:30.740947 16186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:30.741080 16186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:30.741618 16178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:30.748732 16200 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:30.748821 16200 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: loading cluster ID for follower catalog manager: success
I20240501 19:57:30.751993 16200 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: acquiring CA information for follower catalog manager: success
I20240501 19:57:30.755918 16200 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:30.759615 15800 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]
I20240501 19:57:30.783666 16206 consensus_queue.cc:1035] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.788043 16141 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:56459; Asking this server to re-register.
I20240501 19:57:30.788959 15512 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:30.790567 16141 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:30.801719 16206 consensus_queue.cc:1035] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:30.804960 16139 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:59359; Asking this server to re-register.
I20240501 19:57:30.805593 15648 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:30.806481 16141 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:39261; Asking this server to re-register.
I20240501 19:57:30.807569 15374 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:30.809311 16139 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:30.812562 16141 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:30.824070 16152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:30.864293 16186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:30.864485 16186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:30.864692 16213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:30.864765 16213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:30.866727 16211 mvcc.cc:205] Tried to move back new op lower bound from 7022974773235621888 to 7022974770271096832. Current Snapshot: MvccSnapshot[applied={T|T < 7022974773235621888}]
I20240501 19:57:30.955842 15797 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]
I20240501 19:57:31.122336 15801 catalog_manager.cc:2145] created HMS catalog entry for table [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:31.124583 16103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:31.124747 16103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:31.127877 15801 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 27) took 1333 ms. Trace:
I20240501 19:57:31.127996 15801 rpcz_store.cc:276] 0501 19:57:29.794272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.797071 (+ 2799us) service_pool.cc:225] Handling call
0501 19:57:30.473371 (+676300us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.473939 (+ 568us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:31.122323 (+648384us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:31.127520 (+ 5197us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:31.127631 (+ 111us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:31.127837 (+ 206us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:31.123262 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:31.123375 (+ 113us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:31.123380 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:31.123382 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:31.123473 (+ 91us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:31.123485 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:31.123488 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:31.123561 (+ 73us) tablet.cc:699] Row locks acquired
0501 19:57:31.123563 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:31.123598 (+ 35us) write_op.cc:270] Start()
0501 19:57:31.123625 (+ 27us) write_op.cc:276] Timestamp: P: 1714593451123535 usec, L: 0
0501 19:57:31.123627 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:31.123811 (+ 184us) log.cc:842] Serialized 432 byte log entry
0501 19:57:31.126676 (+ 2865us) op_driver.cc:444] REPLICATION: finished
0501 19:57:31.126953 (+ 277us) write_op.cc:301] APPLY: starting
0501 19:57:31.126970 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:31.127002 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:31.127020 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:31.127110 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:31.127143 (+ 33us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:31.127149 (+ 6us) write_op.cc:312] APPLY: finished
0501 19:57:31.127219 (+ 70us) log.cc:842] Serialized 28 byte log entry
0501 19:57:31.127430 (+ 211us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.127446 (+ 16us) write_op.cc:446] Released schema lock
0501 19:57:31.127453 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":481679,"Hive Metastore.run_cpu_time_us":387,"Hive Metastore.run_wall_time_us":166339,"spinlock_wait_cycles":16768,"child_traces":[["op",{"apply.queue_time_us":219,"leader-initialization.queue_time_us":223,"leader-initialization.run_cpu_time_us":11,"leader-initialization.run_wall_time_us":11,"mutex_wait_us":1,"num_ops":2,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":1024,"raft.queue_time_us":471,"raft.run_cpu_time_us":789,"raft.run_wall_time_us":787,"replication_time_us":3029,"spinlock_wait_cycles":1408,"thread_start_us":413,"threads_started":5}]]}
I20240501 19:57:31.136726 15440 tablet_service.cc:1467] Processing CreateTablet for tablet be78b5885c0a43ffa72eebb60579b578 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.137135 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78b5885c0a43ffa72eebb60579b578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.137130 15801 catalog_manager.cc:5484] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:31.137579 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.137742 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:31.137785 15578 tablet_service.cc:1467] Processing CreateTablet for tablet be78b5885c0a43ffa72eebb60579b578 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.137820 15801 catalog_manager.cc:5484] T f8c7c7763c08463ba9d1056d39988b4d P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } }
I20240501 19:57:31.138011 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.138109 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78b5885c0a43ffa72eebb60579b578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.139487 15303 tablet_service.cc:1467] Processing CreateTablet for tablet be78b5885c0a43ffa72eebb60579b578 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.139787 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be78b5885c0a43ffa72eebb60579b578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.142948 16226 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:31.144191 16226 tablet_bootstrap.cc:654] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.145526 16226 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:31.145627 16226 ts_tablet_manager.cc:1367] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:31.146160 16226 raft_consensus.cc:348] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.146296 16226 raft_consensus.cc:374] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.146327 16226 raft_consensus.cc:729] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.146461 16226 consensus_queue.cc:260] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.146726 16226 ts_tablet_manager.cc:1398] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:31.154582 15801 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42] alter complete (version 1)
I20240501 19:57:31.154810 16227 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:31.156723 16227 tablet_bootstrap.cc:654] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.157100 16228 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:31.158198 16228 tablet_bootstrap.cc:654] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.159982 15801 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992] alter complete (version 2)
W20240501 19:57:31.160313 15801 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet fe8ac073d20b45138610f9f4ffad1138 consensus state has no leader
I20240501 19:57:31.161531 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:31.161767 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:40968
I20240501 19:57:31.162541 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.162678 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
W20240501 19:57:31.162716 15787 catalog_manager.cc:4797] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): delete failed for tablet 34f6808094c74a4e8b3b7d092d863042 because the tablet was not found. No further retry: Not found: Tablet not found: 34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:31.162778 16227 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:31.162868 16227 ts_tablet_manager.cc:1367] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
W20240501 19:57:31.162909 15787 catalog_manager.cc:4797] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): delete failed for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d because the tablet was not found. No further retry: Not found: Tablet not found: 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
I20240501 19:57:31.162802 15801 catalog_manager.cc:5484] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:31.163250 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.163537 16227 raft_consensus.cc:348] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.163693 16227 raft_consensus.cc:374] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.163763 16227 raft_consensus.cc:729] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.163935 16227 consensus_queue.cc:260] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.164265 16227 ts_tablet_manager.cc:1398] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:31.166740 16228 tablet_bootstrap.cc:492] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:31.166944 16228 ts_tablet_manager.cc:1367] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:31.167578 16228 raft_consensus.cc:348] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.167775 16228 raft_consensus.cc:374] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.167881 16228 raft_consensus.cc:729] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.169051 16228 consensus_queue.cc:260] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.169452 16228 ts_tablet_manager.cc:1398] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:31.170902 15440 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49454
I20240501 19:57:31.171854 15440 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:49454
W20240501 19:57:31.172338 15790 catalog_manager.cc:4797] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): delete failed for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d because the tablet was not found. No further retry: Not found: Tablet not found: 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
W20240501 19:57:31.172534 15790 catalog_manager.cc:4797] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): delete failed for tablet 34f6808094c74a4e8b3b7d092d863042 because the tablet was not found. No further retry: Not found: Tablet not found: 34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:31.173418 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.173547 15801 catalog_manager.cc:5484] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:31.173750 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:31.173919 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.179458 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:56694
I20240501 19:57:31.179591 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f6808094c74a4e8b3b7d092d863042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:30 UTC) from {username='slave'} at 127.0.0.1:56694
W20240501 19:57:31.180087 15789 catalog_manager.cc:4797] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): delete failed for tablet 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d because the tablet was not found. No further retry: Not found: Tablet not found: 0d3b9f8b8fe34fe49c6cc0cf0ca2e07d
W20240501 19:57:31.180346 15789 catalog_manager.cc:4797] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): delete failed for tablet 34f6808094c74a4e8b3b7d092d863042 because the tablet was not found. No further retry: Not found: Tablet not found: 34f6808094c74a4e8b3b7d092d863042
I20240501 19:57:31.180527 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.180796 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:31.180974 15801 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.183250 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.183527 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:31.183647 15801 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.185235 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]). Expected version 1 got 0
I20240501 19:57:31.185372 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.185441 15801 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]). Expected version 1 got 0
I20240501 19:57:31.186962 15801 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_206463f213f945b2a74cbbcddbada5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:31.191809 15741 raft_consensus.cc:482] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:31.191964 15741 raft_consensus.cc:504] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.192564 15741 leader_election.cc:288] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.193092 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78b5885c0a43ffa72eebb60579b578" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:31.193220 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78b5885c0a43ffa72eebb60579b578" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:31.193292 15598 raft_consensus.cc:2454] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.193367 15325 raft_consensus.cc:2454] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.193682 15399 leader_election.cc:302] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:31.193895 15741 raft_consensus.cc:2781] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:31.193956 15741 raft_consensus.cc:482] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:31.193985 15741 raft_consensus.cc:3037] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.207615 16233 tablet.cc:1722] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.207976 16234 tablet.cc:1722] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.209651 16235 tablet.cc:1722] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.218817 15741 raft_consensus.cc:504] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.219425 15741 leader_election.cc:288] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.220265 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78b5885c0a43ffa72eebb60579b578" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:31.220429 15325 raft_consensus.cc:3037] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.223954 15325 raft_consensus.cc:2454] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.224332 15397 leader_election.cc:302] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.225086 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be78b5885c0a43ffa72eebb60579b578" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:31.225230 15598 raft_consensus.cc:3037] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.226372 15741 raft_consensus.cc:2781] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:31.226500 15741 raft_consensus.cc:686] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:31.226645 15741 consensus_queue.cc:237] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.227057 15803 catalog_manager.cc:2145] created HMS catalog entry for table [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:31.231969 15803 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 33) took 1437 ms. Trace:
I20240501 19:57:31.232069 15803 rpcz_store.cc:276] 0501 19:57:29.794704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.805285 (+ 10581us) service_pool.cc:225] Handling call
0501 19:57:29.809571 (+ 4286us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x55f241f1c0d4. stack: 00007f938befe903 00007f938befeef5 00007f938ba8089d 000055f240fc336e 000055f240fc33c1 00007f938e8408eb 00007f938e83f0f7 00007f938daadbab 00007f938f9848e5 00007f938fa49bd7 00007f938d5a6f01 00007f938d5b0125 00007f938cb64411 00007f938cb63cc5 00007f938cb66b18 00007f938cb65352
0501 19:57:30.473867 (+664296us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.474155 (+ 288us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:31.227049 (+752894us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:31.231774 (+ 4725us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:31.231873 (+ 99us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:31.231951 (+ 78us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:31.227589 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:31.227697 (+ 108us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:31.227701 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:31.227703 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:31.227773 (+ 70us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:31.227785 (+ 12us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:31.227788 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:31.227846 (+ 58us) tablet.cc:699] Row locks acquired
0501 19:57:31.227874 (+ 28us) write_op.cc:260] PREPARE: finished
0501 19:57:31.227905 (+ 31us) write_op.cc:270] Start()
0501 19:57:31.227932 (+ 27us) write_op.cc:276] Timestamp: P: 1714593451227882 usec, L: 0
0501 19:57:31.227934 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:31.228076 (+ 142us) log.cc:842] Serialized 432 byte log entry
0501 19:57:31.231201 (+ 3125us) op_driver.cc:444] REPLICATION: finished
0501 19:57:31.231300 (+ 99us) write_op.cc:301] APPLY: starting
0501 19:57:31.231318 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:31.231352 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:31.231367 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:31.231462 (+ 95us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:31.231493 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:31.231498 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:31.231565 (+ 67us) log.cc:842] Serialized 28 byte log entry
0501 19:57:31.231618 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.231635 (+ 17us) write_op.cc:446] Released schema lock
0501 19:57:31.231641 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:31.231689 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.231692 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":647996,"Hive Metastore.run_cpu_time_us":220,"Hive Metastore.run_wall_time_us":104562,"spinlock_wait_cycles":9466880,"child_traces":[["op",{"apply.queue_time_us":54,"apply.run_cpu_time_us":438,"apply.run_wall_time_us":438,"num_ops":2,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":588,"prepare.run_wall_time_us":588,"raft.queue_time_us":383,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":420,"replication_time_us":3244}]]}
I20240501 19:57:31.235114 15803 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_abc8d72a6cc74062a62861c18c185a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:31.240401 15440 tablet_service.cc:1467] Processing CreateTablet for tablet e57997f8a3a742ed86ca9dca11161071 (DEFAULT_TABLE table=default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.240435 15579 tablet_service.cc:1467] Processing CreateTablet for tablet e57997f8a3a742ed86ca9dca11161071 (DEFAULT_TABLE table=default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.240767 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57997f8a3a742ed86ca9dca11161071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.240787 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57997f8a3a742ed86ca9dca11161071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.242755 15303 tablet_service.cc:1467] Processing CreateTablet for tablet e57997f8a3a742ed86ca9dca11161071 (DEFAULT_TABLE table=default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.243098 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57997f8a3a742ed86ca9dca11161071. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.243412 16226 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:31.244626 16226 tablet_bootstrap.cc:654] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.246510 16227 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:31.247651 16227 tablet_bootstrap.cc:654] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.247690 16226 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:31.247798 16226 ts_tablet_manager.cc:1367] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:31.248383 16226 raft_consensus.cc:348] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.248546 16226 raft_consensus.cc:374] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.248594 16226 raft_consensus.cc:729] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.248746 16226 consensus_queue.cc:260] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.249083 16226 ts_tablet_manager.cc:1398] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:31.249231 16227 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:31.249325 16227 ts_tablet_manager.cc:1367] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:31.249819 16227 raft_consensus.cc:348] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.249935 16227 raft_consensus.cc:374] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.249974 16227 raft_consensus.cc:729] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.250118 16227 consensus_queue.cc:260] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.250418 16227 ts_tablet_manager.cc:1398] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:31.322661 15598 raft_consensus.cc:2454] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.323709 16228 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:31.324956 16228 tablet_bootstrap.cc:654] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.326184 16228 tablet_bootstrap.cc:492] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:31.326329 16228 ts_tablet_manager.cc:1367] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:31.326786 16228 raft_consensus.cc:348] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.326894 16228 raft_consensus.cc:374] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.326928 16228 raft_consensus.cc:729] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.327104 16228 consensus_queue.cc:260] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.327399 16228 ts_tablet_manager.cc:1398] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:31.434540 15741 raft_consensus.cc:482] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:31.434675 15741 raft_consensus.cc:504] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.435117 15741 leader_election.cc:288] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.435655 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57997f8a3a742ed86ca9dca11161071" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:31.435832 15325 raft_consensus.cc:2454] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.436167 15397 leader_election.cc:302] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.436254 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57997f8a3a742ed86ca9dca11161071" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:31.436381 15741 raft_consensus.cc:2781] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:31.436410 15598 raft_consensus.cc:2454] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.436450 15741 raft_consensus.cc:482] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:31.436511 15741 raft_consensus.cc:3037] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.522845 15741 raft_consensus.cc:504] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.523346 15741 leader_election.cc:288] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.523797 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57997f8a3a742ed86ca9dca11161071" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:31.523831 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57997f8a3a742ed86ca9dca11161071" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:31.523948 15325 raft_consensus.cc:3037] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.523948 15598 raft_consensus.cc:3037] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.524995 15598 raft_consensus.cc:2454] T e57997f8a3a742ed86ca9dca11161071 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.524995 15325 raft_consensus.cc:2454] T e57997f8a3a742ed86ca9dca11161071 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.525454 15397 leader_election.cc:302] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.525674 15741 raft_consensus.cc:2781] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:31.525784 15741 raft_consensus.cc:686] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:31.525943 15741 consensus_queue.cc:237] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.528163 15802 catalog_manager.cc:2145] created HMS catalog entry for table [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:31.532598 15802 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 35) took 1737 ms. Trace:
I20240501 19:57:31.532697 15802 rpcz_store.cc:276] 0501 19:57:29.794811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.807507 (+ 12696us) service_pool.cc:225] Handling call
0501 19:57:29.813569 (+ 6062us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x55f241f1c0d4. stack: 00007f938befe903 00007f938befeef5 00007f938ba8089d 000055f240fc336e 000055f240fc33c1 00007f938e8408eb 00007f938e83f0f7 00007f938daadbab 00007f938f9848e5 00007f938fa49bd7 00007f938d5a6f01 00007f938d5b0125 00007f938cb64411 00007f938cb63cc5 00007f938cb66b18 00007f938cb65352
0501 19:57:30.474192 (+660623us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.474462 (+ 270us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:31.528153 (+1053691us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:31.532396 (+ 4243us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:31.532494 (+ 98us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:31.532579 (+ 85us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:31.528715 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:31.528806 (+ 91us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:31.528810 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:31.528811 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:31.528906 (+ 95us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:31.528916 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:31.528919 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:31.528982 (+ 63us) tablet.cc:699] Row locks acquired
0501 19:57:31.528984 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:31.529015 (+ 31us) write_op.cc:270] Start()
0501 19:57:31.529041 (+ 26us) write_op.cc:276] Timestamp: P: 1714593451528964 usec, L: 0
0501 19:57:31.529043 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:31.529182 (+ 139us) log.cc:842] Serialized 432 byte log entry
0501 19:57:31.531444 (+ 2262us) op_driver.cc:444] REPLICATION: finished
0501 19:57:31.531530 (+ 86us) write_op.cc:301] APPLY: starting
0501 19:57:31.531548 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:31.531585 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:31.531600 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:31.531680 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:31.531717 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:31.531724 (+ 7us) write_op.cc:312] APPLY: finished
0501 19:57:31.531785 (+ 61us) log.cc:842] Serialized 28 byte log entry
0501 19:57:31.531844 (+ 59us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.531860 (+ 16us) write_op.cc:446] Released schema lock
0501 19:57:31.531865 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:31.531910 (+ 45us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.531912 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":752313,"Hive Metastore.run_cpu_time_us":248,"Hive Metastore.run_wall_time_us":301113,"spinlock_wait_cycles":6214144,"child_traces":[["op",{"apply.queue_time_us":30,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":427,"num_ops":2,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":565,"prepare.run_wall_time_us":566,"raft.queue_time_us":306,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":343,"replication_time_us":2380}]]}
I20240501 19:57:31.533754 15802 catalog_manager.cc:5484] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:31.547996 15440 tablet_service.cc:1467] Processing CreateTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.548347 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e268796c5da43fa9d65a3b9d0a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.548631 15802 catalog_manager.cc:5484] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:31.549240 15303 tablet_service.cc:1467] Processing CreateTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.549510 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e268796c5da43fa9d65a3b9d0a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.550737 16226 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:31.551757 16227 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:31.552513 16226 tablet_bootstrap.cc:654] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.552716 16227 tablet_bootstrap.cc:654] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.555003 16227 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:31.555112 16227 ts_tablet_manager.cc:1367] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:31.555343 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.555660 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e268796c5da43fa9d65a3b9d0a444d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.555601 16227 raft_consensus.cc:348] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.555711 16227 raft_consensus.cc:374] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.555742 16227 raft_consensus.cc:729] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.555862 16227 consensus_queue.cc:260] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.556138 16227 ts_tablet_manager.cc:1398] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:31.558301 16240 raft_consensus.cc:482] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:31.558436 16228 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:31.558400 16240 raft_consensus.cc:504] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.558786 16240 leader_election.cc:288] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:31.559319 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:31.559386 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:31.559463 16228 tablet_bootstrap.cc:654] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
W20240501 19:57:31.559844 15262 leader_election.cc:341] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): Illegal state: must be running to vote when last-logged opid is not known
W20240501 19:57:31.560318 15261 leader_election.cc:341] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:31.560397 15261 leader_election.cc:302] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d; no voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:31.560660 16240 raft_consensus.cc:2726] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:31.561028 16228 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:31.561106 16228 ts_tablet_manager.cc:1367] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:31.564054 16228 raft_consensus.cc:348] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.564185 16228 raft_consensus.cc:374] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.564237 16228 raft_consensus.cc:729] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.565979 15802 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:31.566291 16228 consensus_queue.cc:260] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.566757 16228 ts_tablet_manager.cc:1398] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:31.567385 15802 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_b0d6857034b54b258f3f6d577cd53e13" } new_table_name: "default.Table_0e2de9c19f564d149e60a8bac7d8ae48" modify_external_catalogs: true
I20240501 19:57:31.567606 16226 tablet_bootstrap.cc:492] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:31.567711 16226 ts_tablet_manager.cc:1367] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240501 19:57:31.568293 16226 raft_consensus.cc:348] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.568403 16226 raft_consensus.cc:374] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.568436 16226 raft_consensus.cc:729] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.568558 16226 consensus_queue.cc:260] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.568859 16226 ts_tablet_manager.cc:1398] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:31.605628 15741 raft_consensus.cc:482] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:31.605819 15741 raft_consensus.cc:504] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.606325 15741 leader_election.cc:288] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.606865 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:31.606865 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:31.607028 15325 raft_consensus.cc:2454] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.607028 15598 raft_consensus.cc:2454] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.607385 15397 leader_election.cc:302] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.607751 15741 raft_consensus.cc:2781] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:31.607811 15741 raft_consensus.cc:482] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:31.607872 15741 raft_consensus.cc:3037] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.625340 15741 raft_consensus.cc:504] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.625875 15741 leader_election.cc:288] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.626281 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:31.626430 15325 raft_consensus.cc:3037] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.626406 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e268796c5da43fa9d65a3b9d0a444d5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:31.626538 15598 raft_consensus.cc:3037] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.627487 15325 raft_consensus.cc:2454] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.627525 15598 raft_consensus.cc:2454] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.627864 15399 leader_election.cc:302] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:31.628232 15741 raft_consensus.cc:2781] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:31.628327 15741 raft_consensus.cc:686] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:31.628482 15741 consensus_queue.cc:237] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.629145 15796 catalog_manager.cc:2145] created HMS catalog entry for table [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:31.633152 15796 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 32) took 1838 ms. Trace:
I20240501 19:57:31.633253 15796 rpcz_store.cc:276] 0501 19:57:29.794575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.805279 (+ 10704us) service_pool.cc:225] Handling call
0501 19:57:29.809223 (+ 3944us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x55f241f1c0d4. stack: 00007f938befe903 00007f938befeef5 00007f938ba8089d 000055f240fc336e 000055f240fc33c1 00007f938e8408eb 00007f938e83f0f7 00007f938daadbab 00007f938f9848e5 00007f938fa49bd7 00007f938d5a6f01 00007f938d5b0125 00007f938cb64411 00007f938cb63cc5 00007f938cb66b18 00007f938cb65352
0501 19:57:30.474354 (+665131us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.474654 (+ 300us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:31.629134 (+1154480us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:31.632932 (+ 3798us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:31.633051 (+ 119us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:31.633134 (+ 83us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:31.629678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:31.629781 (+ 103us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:31.629786 (+ 5us) write_op.cc:427] Acquired schema lock
0501 19:57:31.629788 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:31.629856 (+ 68us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:31.629866 (+ 10us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:31.629869 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:31.629928 (+ 59us) tablet.cc:699] Row locks acquired
0501 19:57:31.629930 (+ 2us) write_op.cc:260] PREPARE: finished
0501 19:57:31.629959 (+ 29us) write_op.cc:270] Start()
0501 19:57:31.629989 (+ 30us) write_op.cc:276] Timestamp: P: 1714593451629899 usec, L: 0
0501 19:57:31.629991 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:31.630122 (+ 131us) log.cc:842] Serialized 432 byte log entry
0501 19:57:31.632304 (+ 2182us) op_driver.cc:444] REPLICATION: finished
0501 19:57:31.632502 (+ 198us) write_op.cc:301] APPLY: starting
0501 19:57:31.632521 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:31.632555 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:31.632572 (+ 17us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:31.632664 (+ 92us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:31.632699 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:31.632704 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:31.632762 (+ 58us) log.cc:842] Serialized 28 byte log entry
0501 19:57:31.632818 (+ 56us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.632866 (+ 48us) write_op.cc:446] Released schema lock
0501 19:57:31.632871 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:31.632919 (+ 48us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:31.632921 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1053267,"Hive Metastore.run_cpu_time_us":225,"Hive Metastore.run_wall_time_us":100902,"spinlock_wait_cycles":8216960,"child_traces":[["op",{"apply.queue_time_us":156,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":464,"num_ops":2,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":540,"prepare.run_wall_time_us":541,"raft.queue_time_us":322,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":380,"replication_time_us":2294}]]}
I20240501 19:57:31.641029 15796 catalog_manager.cc:5484] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:31.641767 15440 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4a80a5d2f848b78741b35319aa54f5 (DEFAULT_TABLE table=default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.642120 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4a80a5d2f848b78741b35319aa54f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.643385 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4a80a5d2f848b78741b35319aa54f5 (DEFAULT_TABLE table=default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.643419 15303 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4a80a5d2f848b78741b35319aa54f5 (DEFAULT_TABLE table=default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:31.643680 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4a80a5d2f848b78741b35319aa54f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.643683 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4a80a5d2f848b78741b35319aa54f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:31.644546 16226 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:31.645556 16226 tablet_bootstrap.cc:654] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.646112 16227 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:31.646248 16228 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:31.647055 16227 tablet_bootstrap.cc:654] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.647163 16228 tablet_bootstrap.cc:654] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:31.649154 15796 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_ea70d92de3fc40b4b3e8c54d6e5b6899" } new_table_name: "default.Table_9254e341432a42c29665f212b2913ff6" modify_external_catalogs: true
I20240501 19:57:31.650319 16226 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:31.650421 16226 ts_tablet_manager.cc:1367] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:31.650900 16226 raft_consensus.cc:348] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.651031 16226 raft_consensus.cc:374] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.651072 16226 raft_consensus.cc:729] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.651206 16226 consensus_queue.cc:260] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.651366 16227 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:31.651464 16227 ts_tablet_manager.cc:1367] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:57:31.651662 16226 ts_tablet_manager.cc:1398] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:31.651717 16228 tablet_bootstrap.cc:492] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:31.651809 16228 ts_tablet_manager.cc:1367] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:31.651922 16227 raft_consensus.cc:348] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.652029 16227 raft_consensus.cc:374] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.652060 16227 raft_consensus.cc:729] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.652189 16227 consensus_queue.cc:260] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.652261 16228 raft_consensus.cc:348] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.652367 16228 raft_consensus.cc:374] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:31.652397 16228 raft_consensus.cc:729] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:31.652467 16227 ts_tablet_manager.cc:1398] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:31.652523 16228 consensus_queue.cc:260] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.655232 16228 ts_tablet_manager.cc:1398] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:31.662029 16235 tablet.cc:1722] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.662942 16233 tablet.cc:1722] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.663821 16234 tablet.cc:1722] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.695590 15741 raft_consensus.cc:482] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:31.695730 15741 raft_consensus.cc:504] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.696372 15741 leader_election.cc:288] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.696926 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4a80a5d2f848b78741b35319aa54f5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:31.696945 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4a80a5d2f848b78741b35319aa54f5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:31.697090 15599 raft_consensus.cc:2454] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.697073 15325 raft_consensus.cc:2454] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:31.697429 15397 leader_election.cc:302] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.697736 15741 raft_consensus.cc:2781] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:31.697789 15741 raft_consensus.cc:482] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:31.697840 15741 raft_consensus.cc:3037] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.712278 16257 tablet.cc:1722] T ec1ef46612cd4545a753bd4700897535 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.712442 16258 tablet.cc:1722] T ec1ef46612cd4545a753bd4700897535 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.712983 16259 tablet.cc:1722] T ec1ef46612cd4545a753bd4700897535 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:31.727937 15741 raft_consensus.cc:504] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.728490 15741 leader_election.cc:288] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:31.729323 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4a80a5d2f848b78741b35319aa54f5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:31.729473 15599 raft_consensus.cc:3037] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.730417 15325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4a80a5d2f848b78741b35319aa54f5" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:31.730564 15325 raft_consensus.cc:3037] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.773408 15741 consensus_queue.cc:1035] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.778474 16232 consensus_queue.cc:1035] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.822280 15325 raft_consensus.cc:2454] T 4d4a80a5d2f848b78741b35319aa54f5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.823064 15599 raft_consensus.cc:2454] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:31.823612 15397 leader_election.cc:302] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:31.823863 15741 raft_consensus.cc:2781] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:31.823952 15741 raft_consensus.cc:686] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:31.824121 15741 consensus_queue.cc:237] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:31.944229 15741 consensus_queue.cc:1035] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.948683 16232 consensus_queue.cc:1035] T e57997f8a3a742ed86ca9dca11161071 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:32.113837 15741 consensus_queue.cc:1035] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:32.118511 15741 consensus_queue.cc:1035] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:32.147688 16267 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:32.147838 16267 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.148375 16267 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:32.148813 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:32.150646 16270 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:32.150820 16270 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.152510 15003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:32.152655 15003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d8e36923de4d1e8a6de93e73c84658 in term 2.
I20240501 19:57:32.152957 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
I20240501 19:57:32.153012 16128 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:32.153209 16152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f177158f524f0183f6eec721ea83e4 in term 2.
I20240501 19:57:32.153228 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:32.153486 16271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:57:32.153493 14977 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763
I20240501 19:57:32.153545 16271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:32.153571 16270 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:32.153607 16271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:32.153729 16267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:57:32.153779 16267 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:32.153815 16267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:32.218811 16271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.218811 16267 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.219365 16267 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 3 election: Requested vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:32.219377 16271 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 3 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:32.219938 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763"
I20240501 19:57:32.219995 15002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:32.220131 15003 raft_consensus.cc:1226] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Rejecting Update request from peer 62106a6dbb3846df9d3b29ab99fb3763 for earlier term 2. Current term is 3. Ops: []
I20240501 19:57:32.221154 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658"
I20240501 19:57:32.221343 16152 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e9f177158f524f0183f6eec721ea83e4 in current term 3: Already voted for candidate 72d8e36923de4d1e8a6de93e73c84658 in this term.
I20240501 19:57:32.221707 14977 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:32.222216 16272 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 }, Status: INVALID_TERM, Last received: 2.125, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240501 19:57:32.222600 16272 raft_consensus.cc:3032] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:32.222659 16272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Running, Role: LEADER
I20240501 19:57:32.222837 16272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 2.125, Last appended by leader: 125, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.223152 16272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:32.223628 15804 catalog_manager.cc:2145] created HMS catalog entry for table [id=039a525c7e63417aae51c7af40927118]
W20240501 19:57:32.224306 15804 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
I20240501 19:57:32.225766 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 3 candidate_status { last_received { term: 2 index: 125 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763"
I20240501 19:57:32.226743 16267 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 19:57:32.321283 15807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d8e36923de4d1e8a6de93e73c84658 in term 3.
I20240501 19:57:32.321756 16127 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658; no voters: e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:32.322008 16271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:32.322115 16271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 LEADER]: Becoming Leader. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Running, Role: LEADER
I20240501 19:57:32.322453 16271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 2.125, Last appended by leader: 125, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:32.323521 16270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.323622 16270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:32.323948 16274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:32.325982 16274 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:32.326458 16274 catalog_manager.cc:669] Loaded metadata for table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]
I20240501 19:57:32.326747 16274 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:32.327016 16274 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:32.327293 16274 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:32.327508 16274 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:32.327705 16274 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:32.327911 16274 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:32.328105 16274 catalog_manager.cc:669] Loaded metadata for table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]
I20240501 19:57:32.328347 16274 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:32.328559 16274 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:32.328752 16274 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:32.329058 16274 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:32.329274 16274 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:32.329489 16274 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:32.329774 16274 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:32.329968 16274 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:32.331878 16274 tablet_loader.cc:96] loaded metadata for tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60])
I20240501 19:57:32.332257 16274 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:32.332588 16274 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:32.332921 16274 tablet_loader.cc:96] loaded metadata for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:32.333109 16274 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:32.333360 16274 tablet_loader.cc:96] loaded metadata for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33])
I20240501 19:57:32.333539 16274 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:32.333703 16274 tablet_loader.cc:96] loaded metadata for tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:32.334131 16274 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:32.334301 16274 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:32.334648 16274 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:32.334899 16274 tablet_loader.cc:96] loaded metadata for tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:32.335155 16274 tablet_loader.cc:96] loaded metadata for tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:32.335321 16274 tablet_loader.cc:96] loaded metadata for tablet e57997f8a3a742ed86ca9dca11161071 (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:32.335575 16274 tablet_loader.cc:96] loaded metadata for tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:32.335731 16274 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:32.336042 16274 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:32.336220 16274 tablet_loader.cc:96] loaded metadata for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:32.336369 16274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:32.337313 16274 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:32.337366 16274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:32.338454 16274 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:32.339253 16274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Loaded TSK: 0
I20240501 19:57:32.340044 16274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:32.340873 16274 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:32.341739 16274 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 36
I20240501 19:57:32.350476 16232 consensus_queue.cc:1035] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:32.353585 16139 catalog_manager.cc:5484] T 4d4a80a5d2f848b78741b35319aa54f5 P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } }
I20240501 19:57:32.354588 15741 consensus_queue.cc:1035] T 4d4a80a5d2f848b78741b35319aa54f5 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:32.359287 15003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Refusing update from remote peer 72d8e36923de4d1e8a6de93e73c84658: Log matching property violated. Preceding OpId in replica: term: 2 index: 125. Preceding OpId from leader: term: 3 index: 127. (index mismatch)
I20240501 19:57:32.359352 15807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Refusing update from remote peer 72d8e36923de4d1e8a6de93e73c84658: Log matching property violated. Preceding OpId in replica: term: 2 index: 125. Preceding OpId from leader: term: 3 index: 127. (index mismatch)
I20240501 19:57:32.359746 16271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240501 19:57:32.359938 16270 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 126, Last known committed idx: 125, Time since last communication: 0.000s
I20240501 19:57:32.361207 16267 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.361330 16267 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:32.361548 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d8e36923de4d1e8a6de93e73c84658. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.361665 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:32.362457 16271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.362498 16270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.362576 16271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:32.362596 16270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:32.363302 16273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.363324 16267 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:32.363412 16273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:32.363430 16267 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:32.364784 15648 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
I20240501 19:57:32.365082 15374 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
I20240501 19:57:32.372380 16142 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2] alter complete (version 1)
I20240501 19:57:32.374938 15512 heartbeater.cc:503] Master 127.14.96.61:46321 was elected leader, sending a full tablet report...
I20240501 19:57:32.418916 15805 catalog_manager.cc:2145] created HMS catalog entry for table [id=ca0cb5d4470641eb975fa6f8033d346f]
W20240501 19:57:32.419608 15805 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
I20240501 19:57:32.423399 15859 catalog_manager.cc:2706] Deleting table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33] in response to Hive Metastore notification log event 37
W20240501 19:57:32.424288 15859 catalog_manager.cc:2782] Illegal state: an error occurred while updating the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
W20240501 19:57:32.424451 15859 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 37 DROP_TABLE default.table_83b626e1fcb24923aeb39c18f422d4dc: Illegal state: an error occurred while updating the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
W20240501 19:57:32.424535 15859 hms_notification_log_listener.cc:134] Hive Metastore notification log listener poll failed: Service unavailable: lost leadership while handling Hive Metastore notification log events: an error occurred while updating the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
I20240501 19:57:32.424777 15798 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 37) took 2629 ms. Trace:
I20240501 19:57:32.424815 15800 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40740 (request call id 26) took 2630 ms. Trace:
I20240501 19:57:32.424916 15800 rpcz_store.cc:276] 0501 19:57:29.794189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.797344 (+ 3155us) service_pool.cc:225] Handling call
0501 19:57:30.472332 (+674988us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:32.424791 (+1952459us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":3788,"Hive Metastore.run_cpu_time_us":174,"Hive Metastore.run_wall_time_us":283320,"spinlock_wait_cycles":684928}
I20240501 19:57:32.424857 15798 rpcz_store.cc:276] 0501 19:57:29.794879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:30.504881 (+710002us) service_pool.cc:225] Handling call
0501 19:57:32.424754 (+1919873us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 19:57:32.425107 15801 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_206463f213f945b2a74cbbcddbada5e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:32.425714 15798 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.96.3:60419 (request call id 90) took 770 ms (client timeout 1000 ms). Trace:
W20240501 19:57:32.425704 15800 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.96.2:56505 (request call id 94) took 771 ms (client timeout 1000 ms). Trace:
W20240501 19:57:32.425853 15800 rpcz_store.cc:269] 0501 19:57:31.653872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.425132 (+771260us) service_pool.cc:225] Handling call
0501 19:57:32.425681 (+ 549us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 19:57:32.425951 15803 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_abc8d72a6cc74062a62861c18c185a62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:32.426472 15800 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.14.96.1:50879 (request call id 100) took 770 ms (client timeout 1000 ms). Trace:
I20240501 19:57:32.425684 15797 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40740 (request call id 36) took 2630 ms. Trace:
W20240501 19:57:32.426499 15654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.60:40075)
I20240501 19:57:32.426541 15797 rpcz_store.cc:276] 0501 19:57:29.794845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.813598 (+ 18753us) service_pool.cc:225] Handling call
0501 19:57:30.472938 (+659340us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:32.425673 (+1952735us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":286577,"Hive Metastore.run_cpu_time_us":186,"Hive Metastore.run_wall_time_us":196172}
I20240501 19:57:32.426519 15796 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40740 (request call id 42) took 2631 ms. Trace:
I20240501 19:57:32.426687 15796 rpcz_store.cc:276] 0501 19:57:29.795052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:31.648818 (+1853766us) service_pool.cc:225] Handling call
0501 19:57:32.426414 (+777596us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:32.426510 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.426856 15797 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40740 (request call id 43) took 2631 ms. Trace:
I20240501 19:57:32.426885 15796 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40740 (request call id 44) took 2631 ms. Trace:
I20240501 19:57:32.426899 15797 rpcz_store.cc:276] 0501 19:57:29.795090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.426709 (+2631619us) service_pool.cc:225] Handling call
0501 19:57:32.426846 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 19:57:32.425802 15798 rpcz_store.cc:269] 0501 19:57:31.655550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.425159 (+769609us) service_pool.cc:225] Handling call
0501 19:57:32.425690 (+ 531us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408}
I20240501 19:57:32.427160 15797 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40740 (request call id 45) took 2632 ms. Trace:
I20240501 19:57:32.427194 15798 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 46) took 2631 ms. Trace:
I20240501 19:57:32.426925 15796 rpcz_store.cc:276] 0501 19:57:29.795130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.426788 (+2631658us) service_pool.cc:225] Handling call
0501 19:57:32.426880 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427245 15798 rpcz_store.cc:276] 0501 19:57:29.795220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427060 (+2631840us) service_pool.cc:225] Handling call
0501 19:57:32.427187 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427321 15798 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d85e2ff9f8a4df2b0856b9041dd2919 from {username='slave'} at 127.0.0.1:40740
W20240501 19:57:32.426523 15800 rpcz_store.cc:269] 0501 19:57:31.655723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.425969 (+770246us) service_pool.cc:225] Handling call
0501 19:57:32.426460 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427451 15800 master_service.cc:946] ReplaceTablet: received request to replace tablet e896ddb02b824d6c9c5519c457f537bc from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:32.427485 15798 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40740 (request call id 48) took 2417 ms. Trace:
I20240501 19:57:32.427532 15798 rpcz_store.cc:276] 0501 19:57:30.010103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427313 (+2417210us) service_pool.cc:225] Handling call
0501 19:57:32.427476 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427603 15800 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40740 (request call id 49) took 2416 ms. Trace:
I20240501 19:57:32.427637 15798 master_service.cc:946] ReplaceTablet: received request to replace tablet 75b4c1663db04508b03bb074ed9fc788 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:32.427798 15798 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40740 (request call id 50) took 2415 ms. Trace:
I20240501 19:57:32.427835 15798 rpcz_store.cc:276] 0501 19:57:30.011995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427623 (+2415628us) service_pool.cc:225] Handling call
0501 19:57:32.427788 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427985 15798 master_service.cc:946] ReplaceTablet: received request to replace tablet dce88e9c62204ac8a67be46336f09865 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:32.428143 15798 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40740 (request call id 51) took 2411 ms. Trace:
I20240501 19:57:32.428194 15798 rpcz_store.cc:276] 0501 19:57:30.016865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427948 (+2411083us) service_pool.cc:225] Handling call
0501 19:57:32.428132 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.428390 15798 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40740 (request call id 52) took 1724 ms. Trace:
I20240501 19:57:32.428422 15798 rpcz_store.cc:276] 0501 19:57:30.703980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.428264 (+1724284us) service_pool.cc:225] Handling call
0501 19:57:32.428383 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.428565 15798 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40740 (request call id 53) took 1724 ms. Trace:
I20240501 19:57:32.428591 15798 rpcz_store.cc:276] 0501 19:57:30.704182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.428485 (+1724303us) service_pool.cc:225] Handling call
0501 19:57:32.428560 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.428750 15798 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 54) took 1097 ms. Trace:
I20240501 19:57:32.427321 15796 master_service.cc:946] ReplaceTablet: received request to replace tablet 34f6808094c74a4e8b3b7d092d863042 from {username='slave'} at 127.0.0.1:40740
I20240501 19:57:32.429056 15796 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:40740 (request call id 47) took 2419 ms. Trace:
I20240501 19:57:32.429100 15796 rpcz_store.cc:276] 0501 19:57:30.009874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427301 (+2417427us) service_pool.cc:225] Handling call
0501 19:57:32.429045 (+ 1744us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.429742 15796 rpcz_store.cc:275] Call kudu.master.MasterService.GetMasterRegistration from 127.0.0.1:40768 (request call id 0) took 1648 ms. Trace:
I20240501 19:57:32.429803 15796 rpcz_store.cc:276] 0501 19:57:30.781477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.429546 (+1648069us) service_pool.cc:225] Handling call
0501 19:57:32.429733 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427206 15797 rpcz_store.cc:276] 0501 19:57:29.795165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427033 (+2631868us) service_pool.cc:225] Handling call
0501 19:57:32.427150 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240501 19:57:32.427639 15800 rpcz_store.cc:276] 0501 19:57:30.011160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.427436 (+2416276us) service_pool.cc:225] Handling call
0501 19:57:32.427596 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 19:57:32.433337 15654 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.60:40075)
I20240501 19:57:32.428791 15798 rpcz_store.cc:276] 0501 19:57:31.331243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.428644 (+1097401us) service_pool.cc:225] Handling call
0501 19:57:32.428741 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240501 19:57:32.433787 15654 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.60:40075)
W20240501 19:57:32.434183 15654 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.60:40075)
I20240501 19:57:32.435603 16100 master_runner.cc:232] Existing masters have differing terms: 2 vs 3
I20240501 19:57:32.435753 16100 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:32.444861 16139 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5" } new_table_name: "default.Table_557eeb62e4e64cc6aeca0b1882dd3909" modify_external_catalogs: true
I20240501 19:57:32.445551 16140 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_d8ca1b5399e14ad98016d79abbc191df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.445878 16137 master_service.cc:946] ReplaceTablet: received request to replace tablet 4d85e2ff9f8a4df2b0856b9041dd2919 from {username='slave'} at 127.0.0.1:37088
I20240501 19:57:32.446805 16138 master_service.cc:946] ReplaceTablet: received request to replace tablet 75b4c1663db04508b03bb074ed9fc788 from {username='slave'} at 127.0.0.1:37088
I20240501 19:57:32.447445 16135 master_service.cc:946] ReplaceTablet: received request to replace tablet dce88e9c62204ac8a67be46336f09865 from {username='slave'} at 127.0.0.1:37088
I20240501 19:57:32.448168 16136 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:32.448298 16136 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:32.448484 16134 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:32.448570 16134 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:32.448674 16133 master_service.cc:946] ReplaceTablet: received request to replace tablet 34f6808094c74a4e8b3b7d092d863042 from {username='slave'} at 127.0.0.1:37088
I20240501 19:57:32.449193 16133 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_a4c90268e9694efab8ddbeccf1c2f618"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.451012 16141 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_0c593340e8374d2d809dda0943eb4dd7" } new_table_name: "default.Table_5481ef3ce5fa4ea2b569d6ac2a569b75" modify_external_catalogs: true
I20240501 19:57:32.451313 16142 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_7dee854303424e5db523b5680176f6c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.455098 16199 catalog_manager.cc:2706] Deleting table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33] in response to Hive Metastore notification log event 37
I20240501 19:57:32.456976 16137 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 4d85e2ff9f8a4df2b0856b9041dd2919
I20240501 19:57:32.458367 16137 catalog_manager.cc:6554] ReplaceTablet: tablet 4d85e2ff9f8a4df2b0856b9041dd2919 deleted and replaced by tablet f8981ca996bc49e2b3a9be9bcdb93bb4
I20240501 19:57:32.458668 16137 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_3fd15f4a5d9f4fe2a7062c4fb1c53392"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.461422 16138 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 75b4c1663db04508b03bb074ed9fc788
I20240501 19:57:32.461745 16138 catalog_manager.cc:6554] ReplaceTablet: tablet 75b4c1663db04508b03bb074ed9fc788 deleted and replaced by tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0
I20240501 19:57:32.461977 16138 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_69448af6833b458b99a6e162b53e46de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.462185 16135 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet dce88e9c62204ac8a67be46336f09865
I20240501 19:57:32.462493 16135 catalog_manager.cc:6554] ReplaceTablet: tablet dce88e9c62204ac8a67be46336f09865 deleted and replaced by tablet ca55116fb0b54eb39a00a3fc41211c8f
I20240501 19:57:32.462679 16135 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_0441f7d080e84fb792251357a57aa031" } new_table_name: "default.Table_43971f18a04d42be9b5f69e32a2031ed" modify_external_catalogs: true
I20240501 19:57:32.467367 16199 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 561867ecbf3c4f3a9b28f91f38ac1f4b
I20240501 19:57:32.468245 16199 catalog_manager.cc:2706] Deleting table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60] in response to Hive Metastore notification log event 38
I20240501 19:57:32.470523 15440 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 with delete_type TABLET_DATA_DELETED (replaced by tablet f8981ca996bc49e2b3a9be9bcdb93bb4) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.470800 15443 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b4c1663db04508b03bb074ed9fc788 with delete_type TABLET_DATA_DELETED (replaced by tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.470907 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet dce88e9c62204ac8a67be46336f09865 with delete_type TABLET_DATA_DELETED (replaced by tablet ca55116fb0b54eb39a00a3fc41211c8f) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.471000 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.471315 15441 tablet_service.cc:1467] Processing CreateTablet for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.471612 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8981ca996bc49e2b3a9be9bcdb93bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.474295 16301 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:32.474539 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b4c1663db04508b03bb074ed9fc788 with delete_type TABLET_DATA_DELETED (replaced by tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.474689 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 with delete_type TABLET_DATA_DELETED (replaced by tablet f8981ca996bc49e2b3a9be9bcdb93bb4) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.474798 15577 tablet_service.cc:1514] Processing DeleteTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.474825 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet dce88e9c62204ac8a67be46336f09865 with delete_type TABLET_DATA_DELETED (replaced by tablet ca55116fb0b54eb39a00a3fc41211c8f) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.474916 15576 tablet_service.cc:1467] Processing CreateTablet for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.475195 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8981ca996bc49e2b3a9be9bcdb93bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.475495 16301 tablet_bootstrap.cc:654] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.476769 16300 tablet_replica.cc:330] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:32.476945 16300 raft_consensus.cc:2229] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:32.477031 16302 tablet_replica.cc:330] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:32.477149 16302 raft_consensus.cc:2229] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.477211 16300 raft_consensus.cc:2258] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.477262 16302 raft_consensus.cc:2258] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.477598 16302 ts_tablet_manager.cc:1875] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.477598 16300 ts_tablet_manager.cc:1875] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.481420 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet dce88e9c62204ac8a67be46336f09865 with delete_type TABLET_DATA_DELETED (replaced by tablet ca55116fb0b54eb39a00a3fc41211c8f) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.481824 15302 tablet_service.cc:1467] Processing CreateTablet for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (DEFAULT_TABLE table=default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.482134 15302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8981ca996bc49e2b3a9be9bcdb93bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.482367 16199 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet 14b4c5836101423ab760756c51a8c72b
I20240501 19:57:32.484447 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b4c5836101423ab760756c51a8c72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.481392 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b4c1663db04508b03bb074ed9fc788 with delete_type TABLET_DATA_DELETED (replaced by tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.481357 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d85e2ff9f8a4df2b0856b9041dd2919 with delete_type TABLET_DATA_DELETED (replaced by tablet f8981ca996bc49e2b3a9be9bcdb93bb4) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.481357 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 561867ecbf3c4f3a9b28f91f38ac1f4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.485025 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b4c5836101423ab760756c51a8c72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.485247 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 14b4c5836101423ab760756c51a8c72b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 19:57:32 UTC) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.486474 16301 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:32.486577 16301 ts_tablet_manager.cc:1367] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:32.486546 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (DEFAULT_TABLE table=default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.486948 15303 tablet_service.cc:1467] Processing CreateTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.487093 16301 raft_consensus.cc:348] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.487201 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca55116fb0b54eb39a00a3fc41211c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.487206 16301 raft_consensus.cc:374] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.487239 16301 raft_consensus.cc:729] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.487358 16301 consensus_queue.cc:260] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.487540 15442 tablet_service.cc:1467] Processing CreateTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.487646 16301 ts_tablet_manager.cc:1398] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:32.488296 15580 tablet_service.cc:1467] Processing CreateTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.488322 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (DEFAULT_TABLE table=default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.488559 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.488802 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (DEFAULT_TABLE table=default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.490324 16304 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:32.491533 16305 tablet_replica.cc:330] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:32.491919 16305 raft_consensus.cc:2229] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:32.492362 16305 raft_consensus.cc:2258] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.492887 16304 tablet_bootstrap.cc:654] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.491770 16306 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:32.494200 16305 ts_tablet_manager.cc:1875] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.495664 16306 tablet_bootstrap.cc:654] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.496558 16304 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:32.496649 16304 ts_tablet_manager.cc:1367] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 19:57:32.496939 16306 tablet_bootstrap.cc:492] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:32.497032 16306 ts_tablet_manager.cc:1367] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:32.497139 16304 raft_consensus.cc:348] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.497249 16304 raft_consensus.cc:374] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.497295 16304 raft_consensus.cc:729] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.497432 16304 consensus_queue.cc:260] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.497475 16306 raft_consensus.cc:348] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.497704 16306 raft_consensus.cc:374] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.497749 16306 raft_consensus.cc:729] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.497979 16306 consensus_queue.cc:260] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.498318 16304 ts_tablet_manager.cc:1398] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:57:32.500416 16306 ts_tablet_manager.cc:1398] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:32.500667 16306 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:32.502507 16306 tablet_bootstrap.cc:654] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.503607 16306 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:32.503685 16306 ts_tablet_manager.cc:1367] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:32.504089 16306 raft_consensus.cc:348] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.504189 16306 raft_consensus.cc:374] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.504215 16306 raft_consensus.cc:729] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.504344 16306 consensus_queue.cc:260] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.504586 16306 ts_tablet_manager.cc:1398] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:32.504788 16306 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:32.505697 16306 tablet_bootstrap.cc:654] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.507251 16306 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:32.507405 16306 ts_tablet_manager.cc:1367] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:32.507839 16306 raft_consensus.cc:348] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.507957 16306 raft_consensus.cc:374] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.508291 16306 raft_consensus.cc:729] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.508430 16306 consensus_queue.cc:260] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.508688 16306 ts_tablet_manager.cc:1398] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:32.516450 16276 raft_consensus.cc:482] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:32.516579 16276 raft_consensus.cc:504] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.517045 16276 leader_election.cc:288] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.517428 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8981ca996bc49e2b3a9be9bcdb93bb4" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:32.517505 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8981ca996bc49e2b3a9be9bcdb93bb4" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:32.517568 15326 raft_consensus.cc:2454] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:32.517659 15599 raft_consensus.cc:2454] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:32.517915 15397 leader_election.cc:302] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:32.518060 16276 raft_consensus.cc:2781] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:32.518121 16276 raft_consensus.cc:482] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:32.518172 16276 raft_consensus.cc:3037] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.520277 16276 raft_consensus.cc:504] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.520629 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.520651 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.520684 16276 leader_election.cc:288] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.520870 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca55116fb0b54eb39a00a3fc41211c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.520949 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca55116fb0b54eb39a00a3fc41211c8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.523018 16301 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:32.524103 16301 tablet_bootstrap.cc:654] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.524647 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8981ca996bc49e2b3a9be9bcdb93bb4" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:32.524777 15599 raft_consensus.cc:3037] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.525383 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8981ca996bc49e2b3a9be9bcdb93bb4" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:32.525513 15326 raft_consensus.cc:3037] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.525730 16304 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:32.526583 16302 ts_tablet_manager.cc:1888] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:57:32.526638 16302 log.cc:1196] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/75b4c1663db04508b03bb074ed9fc788
I20240501 19:57:32.526764 16304 tablet_bootstrap.cc:654] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.526847 16302 ts_tablet_manager.cc:1909] T 75b4c1663db04508b03bb074ed9fc788 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:32.528630 16304 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:32.528860 16304 ts_tablet_manager.cc:1367] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 19:57:32.529039 16301 tablet_bootstrap.cc:492] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:32.529135 16301 ts_tablet_manager.cc:1367] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:32.529487 16304 raft_consensus.cc:348] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.529595 16304 raft_consensus.cc:374] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.529650 16304 raft_consensus.cc:729] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.529760 16301 raft_consensus.cc:348] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.529809 16304 consensus_queue.cc:260] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.529870 16301 raft_consensus.cc:374] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.529893 16301 raft_consensus.cc:729] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.530011 16301 consensus_queue.cc:260] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.530109 16304 ts_tablet_manager.cc:1398] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:32.530272 16304 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:32.530282 16301 ts_tablet_manager.cc:1398] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:32.530452 16301 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:32.531381 16301 tablet_bootstrap.cc:654] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.531991 16304 tablet_bootstrap.cc:654] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.533346 16301 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:32.533439 16301 ts_tablet_manager.cc:1367] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:32.533627 16304 tablet_bootstrap.cc:492] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:32.533710 16304 ts_tablet_manager.cc:1367] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:57:32.533950 16301 raft_consensus.cc:348] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.534056 16301 raft_consensus.cc:374] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.534090 16301 raft_consensus.cc:729] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.534215 16301 consensus_queue.cc:260] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.534209 16304 raft_consensus.cc:348] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.534303 16304 raft_consensus.cc:374] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.534410 16304 raft_consensus.cc:729] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.534502 16301 ts_tablet_manager.cc:1398] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:32.534651 16304 consensus_queue.cc:260] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.535882 16304 ts_tablet_manager.cc:1398] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:32.543449 16305 ts_tablet_manager.cc:1888] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:32.543606 16305 log.cc:1196] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/dce88e9c62204ac8a67be46336f09865
I20240501 19:57:32.543951 16305 ts_tablet_manager.cc:1909] T dce88e9c62204ac8a67be46336f09865 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:32.600193 16251 raft_consensus.cc:482] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:32.600345 16251 raft_consensus.cc:504] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.600813 16251 leader_election.cc:288] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.601312 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ba8dce2b2f4d4a8a9db68557fdc2d0" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:32.601485 15598 raft_consensus.cc:2454] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:32.601437 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ba8dce2b2f4d4a8a9db68557fdc2d0" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:32.601603 15462 raft_consensus.cc:2454] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:32.601819 15261 leader_election.cc:302] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:32.601991 16251 raft_consensus.cc:2781] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:32.602062 16251 raft_consensus.cc:482] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:32.602106 16251 raft_consensus.cc:3037] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.613934 15326 raft_consensus.cc:2454] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:32.613932 15599 raft_consensus.cc:2454] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:32.614076 16300 ts_tablet_manager.cc:1888] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:32.614166 16300 log.cc:1196] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/4d85e2ff9f8a4df2b0856b9041dd2919
I20240501 19:57:32.614480 16300 ts_tablet_manager.cc:1909] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:32.614957 16302 tablet_replica.cc:330] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:32.615144 16302 raft_consensus.cc:2229] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.615272 16302 raft_consensus.cc:2258] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.615733 16302 ts_tablet_manager.cc:1875] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.615747 16305 tablet_replica.cc:330] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:32.615881 16305 raft_consensus.cc:2229] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:32.616096 16305 raft_consensus.cc:2258] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.616504 16305 ts_tablet_manager.cc:1875] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.616958 16199 catalog_manager.cc:2706] Deleting table default.table_7a325fdcd5224571b886231cf98de803 [id=039a525c7e63417aae51c7af40927118] in response to Hive Metastore notification log event 45
W20240501 19:57:32.617202 16199 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 45 DROP_TABLE default.table_7a325fdcd5224571b886231cf98de803: Not found: the table does not exist: table_id: "039a525c7e63417aae51c7af40927118" table_name: "default.table_7a325fdcd5224571b886231cf98de803"
I20240501 19:57:32.617355 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:32.617682 15397 leader_election.cc:302] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:32.617878 16276 raft_consensus.cc:2781] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:32.618621 16276 raft_consensus.cc:686] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:32.618829 16276 consensus_queue.cc:237] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.618914 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:32.619398 16300 tablet_replica.cc:330] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:32.619529 16300 raft_consensus.cc:2229] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.619635 16300 raft_consensus.cc:2258] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.620007 16300 ts_tablet_manager.cc:1875] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.621325 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]) successfully deleted
W20240501 19:57:32.623689 16140 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_d8ca1b5399e14ad98016d79abbc191df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:32.623976 16133 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a4c90268e9694efab8ddbeccf1c2f618 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:32.624871 16142 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_7dee854303424e5db523b5680176f6c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240501 19:57:32.625535 16137 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:32.626009 16135 master_service.cc:946] ReplaceTablet: received request to replace tablet e896ddb02b824d6c9c5519c457f537bc from {username='slave'} at 127.0.0.1:37088
W20240501 19:57:32.626178 16138 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_69448af6833b458b99a6e162b53e46de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:32.626374 16141 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37088:
name: "default.Table_a82d93e7c3e949e99304ee3a61f2f5e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:32.631206 16135 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Sending DeleteTablet for 3 replicas of tablet e896ddb02b824d6c9c5519c457f537bc
I20240501 19:57:32.631556 16135 catalog_manager.cc:6554] ReplaceTablet: tablet e896ddb02b824d6c9c5519c457f537bc deleted and replaced by tablet 553a91cc4b2d44feb3d30d3bde9a2919
I20240501 19:57:32.632258 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet e896ddb02b824d6c9c5519c457f537bc with delete_type TABLET_DATA_DELETED (replaced by tablet 553a91cc4b2d44feb3d30d3bde9a2919) from {username='slave'} at 127.0.0.1:56702
I20240501 19:57:32.632714 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet e896ddb02b824d6c9c5519c457f537bc with delete_type TABLET_DATA_DELETED (replaced by tablet 553a91cc4b2d44feb3d30d3bde9a2919) from {username='slave'} at 127.0.0.1:49468
I20240501 19:57:32.633024 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet e896ddb02b824d6c9c5519c457f537bc with delete_type TABLET_DATA_DELETED (replaced by tablet 553a91cc4b2d44feb3d30d3bde9a2919) from {username='slave'} at 127.0.0.1:40980
I20240501 19:57:32.637030 16135 catalog_manager.cc:5484] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:32.637431 16135 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:32.637586 16135 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:32.638062 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.638537 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:32.639312 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
W20240501 19:57:32.641624 16135 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 consensus state has no leader
W20240501 19:57:32.641847 16135 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ca55116fb0b54eb39a00a3fc41211c8f consensus state has no leader
I20240501 19:57:32.643465 16135 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:32.643663 16135 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:32.644577 16135 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_b7b25ac22dba40c9aa0a509437741d50" } new_table_name: "default.Table_3846033cb97a4827a54a0f1bcf875141" modify_external_catalogs: true
W20240501 19:57:32.674557 16127 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 consensus state has no leader
I20240501 19:57:32.709470 16251 raft_consensus.cc:504] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.709584 16302 ts_tablet_manager.cc:1888] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:32.709688 16302 log.cc:1196] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/4d85e2ff9f8a4df2b0856b9041dd2919
I20240501 19:57:32.709977 16302 ts_tablet_manager.cc:1909] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:32.710060 16251 leader_election.cc:288] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.710626 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553a91cc4b2d44feb3d30d3bde9a2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.710896 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553a91cc4b2d44feb3d30d3bde9a2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.713142 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 553a91cc4b2d44feb3d30d3bde9a2919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.715552 16304 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:32.716713 16304 tablet_bootstrap.cc:654] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.717429 15804 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=039a525c7e63417aae51c7af40927118]
I20240501 19:57:32.717648 16306 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:32.717775 15804 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 30) took 2923 ms. Trace:
I20240501 19:57:32.717837 15804 rpcz_store.cc:276] 0501 19:57:29.794484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.796275 (+ 1791us) service_pool.cc:225] Handling call
0501 19:57:30.473454 (+677179us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.474775 (+ 1321us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:32.223623 (+1748848us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:32.224134 (+ 511us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:32.224139 (+ 5us) write_op.cc:446] Released schema lock
0501 19:57:32.224376 (+ 237us) catalog_manager.cc:2152] Rolling back HMS table creation
0501 19:57:32.717757 (+493381us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:32.224058 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER)
0501 19:57:32.224071 (+ 13us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:32.224078 (+ 7us) write_op.cc:446] Released schema lock
0501 19:57:32.224080 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1352763,"Hive Metastore.run_cpu_time_us":389,"Hive Metastore.run_wall_time_us":888789,"child_traces":[["op",{}]]}
I20240501 19:57:32.718834 16306 tablet_bootstrap.cc:654] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.719566 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ba8dce2b2f4d4a8a9db68557fdc2d0" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:32.719708 15462 raft_consensus.cc:3037] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.720826 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ba8dce2b2f4d4a8a9db68557fdc2d0" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:32.720984 15599 raft_consensus.cc:3037] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.722039 16301 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:32.723452 16301 tablet_bootstrap.cc:654] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:32.726686 16302 tablet_replica.cc:330] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:32.726855 16302 raft_consensus.cc:2229] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.726971 16302 raft_consensus.cc:2258] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.727396 16302 ts_tablet_manager.cc:1875] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.728600 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]) successfully deleted
I20240501 19:57:32.729485 16301 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:32.729607 16301 ts_tablet_manager.cc:1367] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240501 19:57:32.730134 16301 raft_consensus.cc:348] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.730324 16301 raft_consensus.cc:374] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.730393 16301 raft_consensus.cc:729] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.730604 16301 consensus_queue.cc:260] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.731707 16301 ts_tablet_manager.cc:1398] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:32.735231 16304 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:32.735357 16304 ts_tablet_manager.cc:1367] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20240501 19:57:32.735870 16304 raft_consensus.cc:348] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.735991 16304 raft_consensus.cc:374] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.736021 16304 raft_consensus.cc:729] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.736971 16306 tablet_bootstrap.cc:492] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:32.737082 16306 ts_tablet_manager.cc:1367] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240501 19:57:32.737506 16306 raft_consensus.cc:348] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.737620 16306 raft_consensus.cc:374] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:32.737653 16306 raft_consensus.cc:729] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:32.737788 16306 consensus_queue.cc:260] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.738098 16306 ts_tablet_manager.cc:1398] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:32.738369 16304 consensus_queue.cc:260] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.741580 16304 ts_tablet_manager.cc:1398] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:32.758740 16276 raft_consensus.cc:482] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:32.758941 16276 raft_consensus.cc:504] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.759451 16276 leader_election.cc:288] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.759932 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca55116fb0b54eb39a00a3fc41211c8f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:32.760108 15326 raft_consensus.cc:2454] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:32.760080 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca55116fb0b54eb39a00a3fc41211c8f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:32.760231 15598 raft_consensus.cc:2454] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:32.760565 15399 leader_election.cc:302] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:32.760753 16276 raft_consensus.cc:2781] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:32.760812 16276 raft_consensus.cc:482] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:32.760877 16276 raft_consensus.cc:3037] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.810284 15599 raft_consensus.cc:2454] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:32.810284 15462 raft_consensus.cc:2454] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:32.810436 16305 ts_tablet_manager.cc:1888] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:57:32.810540 16305 log.cc:1196] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/75b4c1663db04508b03bb074ed9fc788
I20240501 19:57:32.810824 16305 ts_tablet_manager.cc:1909] T 75b4c1663db04508b03bb074ed9fc788 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:32.811586 15262 leader_election.cc:302] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:32.811708 16276 raft_consensus.cc:504] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.811838 16251 raft_consensus.cc:2781] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:32.811931 16251 raft_consensus.cc:686] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:32.812065 16251 consensus_queue.cc:237] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.812192 16276 leader_election.cc:288] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:32.812325 16300 ts_tablet_manager.cc:1888] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 19:57:32.812414 16300 log.cc:1196] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/75b4c1663db04508b03bb074ed9fc788
I20240501 19:57:32.812675 16300 ts_tablet_manager.cc:1909] T 75b4c1663db04508b03bb074ed9fc788 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:32.812721 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca55116fb0b54eb39a00a3fc41211c8f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:32.812827 15326 raft_consensus.cc:3037] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.813764 15326 raft_consensus.cc:2454] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:32.814167 15397 leader_election.cc:302] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:32.814141 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca55116fb0b54eb39a00a3fc41211c8f" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:32.814282 15599 raft_consensus.cc:3037] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.815030 16276 raft_consensus.cc:2781] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:32.815137 16276 raft_consensus.cc:686] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:32.815308 16305 tablet_replica.cc:330] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:32.815289 16276 consensus_queue.cc:237] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:32.815467 16305 raft_consensus.cc:2229] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.815570 16305 raft_consensus.cc:2258] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.815644 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:32.815908 16305 ts_tablet_manager.cc:1875] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.835742 16302 ts_tablet_manager.cc:1888] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:32.835842 16302 log.cc:1196] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/561867ecbf3c4f3a9b28f91f38ac1f4b
I20240501 19:57:32.836102 16302 ts_tablet_manager.cc:1909] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:32.868824 16300 tablet_replica.cc:330] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:32.869093 16300 raft_consensus.cc:2229] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.869246 16300 raft_consensus.cc:2258] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.869606 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 75b4c1663db04508b03bb074ed9fc788 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]) successfully deleted
I20240501 19:57:32.869690 16300 ts_tablet_manager.cc:1875] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 19:57:32.891361 15261 consensus_peers.cc:485] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d -> Peer 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): Couldn't send request to peer 7f011d422e164c949a9d54d846b7c432. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.903597 15599 raft_consensus.cc:2454] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:32.906107 16302 tablet_replica.cc:330] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:32.906332 16302 raft_consensus.cc:2229] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.906455 16302 raft_consensus.cc:2258] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.906908 16302 ts_tablet_manager.cc:1875] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.908077 16305 ts_tablet_manager.cc:1888] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:32.908174 16305 log.cc:1196] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/4d85e2ff9f8a4df2b0856b9041dd2919
I20240501 19:57:32.908453 16305 ts_tablet_manager.cc:1909] T 4d85e2ff9f8a4df2b0856b9041dd2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:32.908491 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]) successfully deleted
I20240501 19:57:32.926609 16300 ts_tablet_manager.cc:1888] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:32.926704 16300 log.cc:1196] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/dce88e9c62204ac8a67be46336f09865
I20240501 19:57:32.926916 16300 ts_tablet_manager.cc:1909] T dce88e9c62204ac8a67be46336f09865 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:32.941313 16305 tablet_replica.cc:330] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:32.941403 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 4d85e2ff9f8a4df2b0856b9041dd2919 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]) successfully deleted
I20240501 19:57:32.941521 16305 raft_consensus.cc:2229] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:32.941784 16305 raft_consensus.cc:2258] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.942186 16305 ts_tablet_manager.cc:1875] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:32.953685 16311 raft_consensus.cc:482] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:32.953828 16311 raft_consensus.cc:504] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:32.954252 16311 leader_election.cc:288] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:32.954730 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553a91cc4b2d44feb3d30d3bde9a2919" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:32.954730 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553a91cc4b2d44feb3d30d3bde9a2919" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:32.954885 15462 raft_consensus.cc:2454] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:32.954885 15599 raft_consensus.cc:2454] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:32.955192 15262 leader_election.cc:302] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:32.955368 16311 raft_consensus.cc:2781] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:32.955441 16311 raft_consensus.cc:482] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:32.955482 16311 raft_consensus.cc:3037] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:32.962038 16300 tablet_replica.cc:330] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:32.962141 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:32.962249 16300 raft_consensus.cc:2229] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:32.962375 16300 raft_consensus.cc:2258] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:32.962757 16300 ts_tablet_manager.cc:1875] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.001786 16311 raft_consensus.cc:504] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:33.002283 16311 leader_election.cc:288] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:33.002763 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553a91cc4b2d44feb3d30d3bde9a2919" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:33.002837 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "553a91cc4b2d44feb3d30d3bde9a2919" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:33.002910 15462 raft_consensus.cc:3037] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.002982 15599 raft_consensus.cc:3037] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.003813 15462 raft_consensus.cc:2454] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:33.003902 16305 ts_tablet_manager.cc:1888] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:33.003971 16305 log.cc:1196] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/561867ecbf3c4f3a9b28f91f38ac1f4b
I20240501 19:57:33.004144 15599 raft_consensus.cc:2454] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:33.004210 15262 leader_election.cc:302] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:33.004393 16311 raft_consensus.cc:2781] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:33.004427 16305 ts_tablet_manager.cc:1909] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:33.004485 16311 raft_consensus.cc:686] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:33.004621 16311 consensus_queue.cc:237] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } }
I20240501 19:57:33.004928 16302 ts_tablet_manager.cc:1888] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.005046 16302 log.cc:1196] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/dce88e9c62204ac8a67be46336f09865
I20240501 19:57:33.005208 16300 ts_tablet_manager.cc:1888] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:33.005338 16300 log.cc:1196] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/561867ecbf3c4f3a9b28f91f38ac1f4b
I20240501 19:57:33.005347 16302 ts_tablet_manager.cc:1909] T dce88e9c62204ac8a67be46336f09865 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:33.005656 16300 ts_tablet_manager.cc:1909] T 561867ecbf3c4f3a9b28f91f38ac1f4b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:33.012382 14720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu with pid 14962
I20240501 19:57:33.037086 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]) successfully deleted
I20240501 19:57:33.037449 16305 tablet_replica.cc:330] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:33.037583 16305 raft_consensus.cc:2229] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:33.037688 16305 raft_consensus.cc:2258] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.038022 16305 ts_tablet_manager.cc:1875] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.042964 16300 tablet_replica.cc:330] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:33.043140 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 561867ecbf3c4f3a9b28f91f38ac1f4b (table default.table_83b626e1fcb24923aeb39c18f422d4dc [id=19c24eab74524422b924c3f739db4f33]) successfully deleted
I20240501 19:57:33.043213 16300 raft_consensus.cc:2229] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:33.043339 16300 raft_consensus.cc:2258] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.043702 16300 ts_tablet_manager.cc:1875] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.044855 16302 tablet_replica.cc:330] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:33.045006 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet dce88e9c62204ac8a67be46336f09865 (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]) successfully deleted
I20240501 19:57:33.045066 16302 raft_consensus.cc:2229] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:33.045269 16302 raft_consensus.cc:2258] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.045625 16302 ts_tablet_manager.cc:1875] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.092449 16300 ts_tablet_manager.cc:1888] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.092659 16300 log.cc:1196] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/14b4c5836101423ab760756c51a8c72b
I20240501 19:57:33.093034 16300 ts_tablet_manager.cc:1909] T 14b4c5836101423ab760756c51a8c72b P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:33.094141 16305 ts_tablet_manager.cc:1888] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.094141 16302 ts_tablet_manager.cc:1888] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.094241 16305 log.cc:1196] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/14b4c5836101423ab760756c51a8c72b
I20240501 19:57:33.094239 16302 log.cc:1196] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/14b4c5836101423ab760756c51a8c72b
I20240501 19:57:33.094537 16302 ts_tablet_manager.cc:1909] T 14b4c5836101423ab760756c51a8c72b P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:33.094552 16305 ts_tablet_manager.cc:1909] T 14b4c5836101423ab760756c51a8c72b P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:33.094955 15741 consensus_queue.cc:1035] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.099385 16276 consensus_queue.cc:1035] T f8981ca996bc49e2b3a9be9bcdb93bb4 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:57:33.118821 16128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:33.119817 16128 consensus_peers.cc:485] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 -> Peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Couldn't send request to peer e9f177158f524f0183f6eec721ea83e4. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:33.122622 16300 tablet_replica.cc:330] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:33.122625 16305 tablet_replica.cc:330] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:33.122799 16300 raft_consensus.cc:2229] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:33.122778 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:33.122777 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:33.122857 16305 raft_consensus.cc:2229] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:33.122962 16300 raft_consensus.cc:2258] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.123095 16305 raft_consensus.cc:2258] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.123389 16300 ts_tablet_manager.cc:1875] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.123482 16305 ts_tablet_manager.cc:1875] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.133824 16302 tablet_replica.cc:330] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:33.133998 16302 raft_consensus.cc:2229] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:33.133994 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 14b4c5836101423ab760756c51a8c72b (table default.table_8060d3196acf479897cfe5f6f547dba9 [id=608d4b3e2b584ef8aac50ccdec82ca60]) successfully deleted
I20240501 19:57:33.134150 16302 raft_consensus.cc:2258] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:33.134557 16302 ts_tablet_manager.cc:1875] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:33.144133 16333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:33.144287 16333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:33.144822 16333 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:33.145354 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 4 candidate_status { last_received { term: 3 index: 140 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
W20240501 19:57:33.146488 15789 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:33.146559 15789 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:33.146694 16333 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 19:57:33.188067 15805 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=ca0cb5d4470641eb975fa6f8033d346f]
I20240501 19:57:33.188364 15805 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 31) took 3393 ms. Trace:
I20240501 19:57:33.188457 15805 rpcz_store.cc:276] 0501 19:57:29.794530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:29.796009 (+ 1479us) service_pool.cc:225] Handling call
0501 19:57:30.473583 (+677574us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:30.474835 (+ 1252us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:32.418902 (+1944067us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:32.419552 (+ 650us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:32.419554 (+ 2us) write_op.cc:446] Released schema lock
0501 19:57:32.419694 (+ 140us) catalog_manager.cc:2152] Rolling back HMS table creation
0501 19:57:33.188341 (+768647us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:32.419497 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER)
0501 19:57:32.419505 (+ 8us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:32.419511 (+ 6us) write_op.cc:446] Released schema lock
0501 19:57:32.419514 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":2046249,"Hive Metastore.run_cpu_time_us":429,"Hive Metastore.run_wall_time_us":665884,"spinlock_wait_cycles":279808,"child_traces":[["op",{}]]}
I20240501 19:57:33.188644 16302 ts_tablet_manager.cc:1888] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.188735 16302 log.cc:1196] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/e896ddb02b824d6c9c5519c457f537bc
I20240501 19:57:33.189019 16302 ts_tablet_manager.cc:1909] T e896ddb02b824d6c9c5519c457f537bc P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:33.189314 16300 ts_tablet_manager.cc:1888] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.189378 16305 ts_tablet_manager.cc:1888] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:33.189407 16300 log.cc:1196] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/e896ddb02b824d6c9c5519c457f537bc
I20240501 19:57:33.189498 16305 log.cc:1196] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/e896ddb02b824d6c9c5519c457f537bc
I20240501 19:57:33.189671 16300 ts_tablet_manager.cc:1909] T e896ddb02b824d6c9c5519c457f537bc P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:33.189785 16305 ts_tablet_manager.cc:1909] T e896ddb02b824d6c9c5519c457f537bc P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:33.207584 16128 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:33.207584 16129 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:33.208411 16126 catalog_manager.cc:4830] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): tablet e896ddb02b824d6c9c5519c457f537bc (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]) successfully deleted
I20240501 19:57:33.222806 16139 catalog_manager.cc:3408] renamed table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28] in HMS: new name default.table_557eeb62e4e64cc6aeca0b1882dd3909
I20240501 19:57:33.231344 16276 consensus_queue.cc:1035] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.236199 16276 consensus_queue.cc:1035] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.253931 16311 consensus_queue.cc:1035] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.255474 16311 consensus_queue.cc:1035] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.261735 15799 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:33.262327 15799 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:40740 (request call id 38) took 3467 ms. Trace:
I20240501 19:57:33.262398 15799 rpcz_store.cc:276] 0501 19:57:29.794920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:30.505936 (+711016us) service_pool.cc:225] Handling call
0501 19:57:32.424877 (+1918941us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:33.262310 (+837433us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":763059,"Hive Metastore.run_cpu_time_us":159,"Hive Metastore.run_wall_time_us":73155}
I20240501 19:57:33.275738 16136 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:33.305230 16134 catalog_manager.cc:2682] deleted HMS catalog entry for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:33.309628 15802 catalog_manager.cc:3408] renamed table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527] in HMS: new name default.table_0e2de9c19f564d149e60a8bac7d8ae48
I20240501 19:57:33.311004 15802 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:40740 (request call id 41) took 3515 ms. Trace:
I20240501 19:57:33.311094 15802 rpcz_store.cc:276] 0501 19:57:29.795019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:31.567251 (+1772232us) service_pool.cc:225] Handling call
0501 19:57:32.424830 (+857579us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:33.310320 (+885490us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0501 19:57:33.310645 (+ 325us) catalog_manager.cc:3613] Apply alter schema
0501 19:57:33.310979 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":836197,"Hive Metastore.run_cpu_time_us":684,"Hive Metastore.run_wall_time_us":48179}
19:57:33.321 [HMS - ERROR - pool-6-thread-6] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_d8ca1b5399e14ad98016d79abbc191df already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy25.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
W20240501 19:57:33.323696 16140 catalog_manager.cc:2141] Already present: failed to create HMS catalog entry for table [id=7c4a6a05788445dc93d84e9b765c4588]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_d8ca1b5399e14ad98016d79abbc191df already exists)
I20240501 19:57:33.325281 16140 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:33.325430 16140 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:33.326691 16140 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
W20240501 19:57:33.327034 16140 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 553a91cc4b2d44feb3d30d3bde9a2919 consensus state has no leader
I20240501 19:57:33.327765 16140 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.328719 16140 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:33.330132 16140 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:33.345212 15801 catalog_manager.cc:2145] created HMS catalog entry for table [id=4315c5cbd36245a785ac6e19bb8806b3]
W20240501 19:57:33.345937 15801 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
I20240501 19:57:33.349033 16142 catalog_manager.cc:2145] created HMS catalog entry for table [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:33.361894 16142 catalog_manager.cc:5484] T 89ba8dce2b2f4d4a8a9db68557fdc2d0 P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } }
I20240501 19:57:33.362125 16142 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:33.362219 16142 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.362290 16142 catalog_manager.cc:5484] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2). New cstate: current_term: 1 leader_uuid: "68539c0ecd3e44e080cb53987d4f4c23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } }
I20240501 19:57:33.362422 16142 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:33.365026 15576 tablet_service.cc:1467] Processing CreateTablet for tablet db41bcd8b98a45a3b384f1642ae4396e (DEFAULT_TABLE table=default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.365348 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41bcd8b98a45a3b384f1642ae4396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.368125 15441 tablet_service.cc:1467] Processing CreateTablet for tablet db41bcd8b98a45a3b384f1642ae4396e (DEFAULT_TABLE table=default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.368429 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41bcd8b98a45a3b384f1642ae4396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.370967 15305 tablet_service.cc:1467] Processing CreateTablet for tablet db41bcd8b98a45a3b384f1642ae4396e (DEFAULT_TABLE table=default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.371279 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db41bcd8b98a45a3b384f1642ae4396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.374475 16348 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:33.374514 16349 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:33.375597 16348 tablet_bootstrap.cc:654] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.377451 16348 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:33.377632 16348 ts_tablet_manager.cc:1367] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:33.377691 16142 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:33.377847 16142 catalog_manager.cc:5484] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1). New cstate: current_term: 1 leader_uuid: "0c8dfc11dc7542f583644b9ac747545d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:33.378077 16142 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.378226 16142 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:33.378541 16348 raft_consensus.cc:348] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.378749 16348 raft_consensus.cc:374] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.378834 16348 raft_consensus.cc:729] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.379071 16348 consensus_queue.cc:260] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.381569 16348 ts_tablet_manager.cc:1398] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:33.383970 16349 tablet_bootstrap.cc:654] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.385385 16142 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.385553 16142 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]). Expected version 1 got 0
I20240501 19:57:33.385562 16349 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:33.385643 16142 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]). Expected version 2 got 0
I20240501 19:57:33.385670 16349 ts_tablet_manager.cc:1367] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:33.386171 16349 raft_consensus.cc:348] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.386291 16349 raft_consensus.cc:374] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.386323 16349 raft_consensus.cc:729] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.386474 16349 consensus_queue.cc:260] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.386770 16349 ts_tablet_manager.cc:1398] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:33.387589 16142 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_090e62cd0c6c43eb9c44ada659e01c98" } new_table_name: "default.Table_f0938e2fdba84a2ca4f93e39ab3e7eac" modify_external_catalogs: true
I20240501 19:57:33.388433 16350 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:33.389573 16350 tablet_bootstrap.cc:654] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.391626 16350 tablet_bootstrap.cc:492] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:33.391744 16350 ts_tablet_manager.cc:1367] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:33.392287 16350 raft_consensus.cc:348] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.392400 16350 raft_consensus.cc:374] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.392431 16350 raft_consensus.cc:729] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.392570 16350 consensus_queue.cc:260] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.393589 16350 ts_tablet_manager.cc:1398] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:33.405728 16311 consensus_queue.cc:1035] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.411625 16251 consensus_queue.cc:1035] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.416700 15803 catalog_manager.cc:2145] created HMS catalog entry for table [id=fa3e8422ce5a432fa46773338cc009e3]
W20240501 19:57:33.417227 15803 catalog_manager.cc:2173] Illegal state: an error occurred while writing to the sys-catalog: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER
I20240501 19:57:33.417539 16137 catalog_manager.cc:2145] created HMS catalog entry for table [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:33.433375 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 568a6e167a0d4040aac09233e544c73d (DEFAULT_TABLE table=default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.433624 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 568a6e167a0d4040aac09233e544c73d (DEFAULT_TABLE table=default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.433727 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a6e167a0d4040aac09233e544c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.433878 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a6e167a0d4040aac09233e544c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.435987 16349 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:33.436219 15305 tablet_service.cc:1467] Processing CreateTablet for tablet 568a6e167a0d4040aac09233e544c73d (DEFAULT_TABLE table=default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.436525 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 568a6e167a0d4040aac09233e544c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.437026 16349 tablet_bootstrap.cc:654] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.437672 16348 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:33.438644 16348 tablet_bootstrap.cc:654] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.438772 16350 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:33.439661 16350 tablet_bootstrap.cc:654] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.446473 16349 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:33.446602 16349 ts_tablet_manager.cc:1367] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:57:33.447086 16349 raft_consensus.cc:348] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.447248 16349 raft_consensus.cc:374] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.447292 16349 raft_consensus.cc:729] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.447434 16349 consensus_queue.cc:260] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.447716 16349 ts_tablet_manager.cc:1398] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:33.449579 16348 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:33.449692 16348 ts_tablet_manager.cc:1367] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:33.450189 16348 raft_consensus.cc:348] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.450325 16348 raft_consensus.cc:374] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.450356 16348 raft_consensus.cc:729] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.450472 16348 consensus_queue.cc:260] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.450726 16348 ts_tablet_manager.cc:1398] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:33.453430 16350 tablet_bootstrap.cc:492] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:33.453559 16350 ts_tablet_manager.cc:1367] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:33.454109 16350 raft_consensus.cc:348] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.454226 16350 raft_consensus.cc:374] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.454257 16350 raft_consensus.cc:729] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.454396 16350 consensus_queue.cc:260] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.454694 16350 ts_tablet_manager.cc:1398] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:33.424791 16137 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_8d1020ff0b064f7199ad44d673369f79" } new_table_name: "default.Table_5dc76a7d16884cc59719736ac8692923" modify_external_catalogs: true
I20240501 19:57:33.462070 16361 tablet.cc:1722] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:33.462474 16360 tablet.cc:1722] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:33.463555 16362 tablet.cc:1722] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 19:57:33.471282 16138 catalog_manager.cc:2145] created HMS catalog entry for table [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:33.475584 16138 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:37088 (request call id 13) took 1031 ms. Trace:
I20240501 19:57:33.475699 16138 rpcz_store.cc:276] 0501 19:57:32.444416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.461856 (+ 17440us) service_pool.cc:225] Handling call
0501 19:57:32.626249 (+164393us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:32.626511 (+ 262us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:33.471269 (+844758us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:33.475367 (+ 4098us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:33.475468 (+ 101us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:33.475560 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:33.471777 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:33.471877 (+ 100us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:33.471881 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:33.471882 (+ 1us) tablet.cc:660] Decoding operations
0501 19:57:33.471953 (+ 71us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:33.471964 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:33.471967 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:33.472026 (+ 59us) tablet.cc:699] Row locks acquired
0501 19:57:33.472027 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:33.472062 (+ 35us) write_op.cc:270] Start()
0501 19:57:33.472088 (+ 26us) write_op.cc:276] Timestamp: P: 1714593453472018 usec, L: 0
0501 19:57:33.472090 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:33.472238 (+ 148us) log.cc:842] Serialized 433 byte log entry
0501 19:57:33.474792 (+ 2554us) op_driver.cc:444] REPLICATION: finished
0501 19:57:33.474900 (+ 108us) write_op.cc:301] APPLY: starting
0501 19:57:33.474917 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:33.474952 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:33.474967 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:33.475049 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:33.475081 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:33.475085 (+ 4us) write_op.cc:312] APPLY: finished
0501 19:57:33.475146 (+ 61us) log.cc:842] Serialized 29 byte log entry
0501 19:57:33.475197 (+ 51us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.475211 (+ 14us) write_op.cc:446] Released schema lock
0501 19:57:33.475216 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:33.475258 (+ 42us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.475261 (+ 3us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":790798,"Hive Metastore.run_cpu_time_us":209,"Hive Metastore.run_wall_time_us":53655,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":401,"apply.run_wall_time_us":400,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":549,"raft.queue_time_us":856,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":177,"replication_time_us":2681}]]}
I20240501 19:57:33.477321 16138 catalog_manager.cc:5576] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.478428 16138 catalog_manager.cc:5576] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.479144 16138 catalog_manager.cc:5576] TS 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247) does not have the latest schema for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]). Expected version 1 got 0
I20240501 19:57:33.479503 16138 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:33.479609 16138 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37088:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:33.486574 15441 tablet_service.cc:1467] Processing CreateTablet for tablet be935df388a842d396a805bfc892496d (DEFAULT_TABLE table=default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.486586 15576 tablet_service.cc:1467] Processing CreateTablet for tablet be935df388a842d396a805bfc892496d (DEFAULT_TABLE table=default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.486899 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be935df388a842d396a805bfc892496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.486900 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be935df388a842d396a805bfc892496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.489200 16349 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:33.489200 16348 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:33.490212 16349 tablet_bootstrap.cc:654] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.490212 16348 tablet_bootstrap.cc:654] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.491858 15305 tablet_service.cc:1467] Processing CreateTablet for tablet be935df388a842d396a805bfc892496d (DEFAULT_TABLE table=default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.492214 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be935df388a842d396a805bfc892496d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.496284 16350 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:33.496512 16349 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:33.496605 16349 ts_tablet_manager.cc:1367] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:57:33.497176 16349 raft_consensus.cc:348] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.497295 16349 raft_consensus.cc:374] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.497326 16349 raft_consensus.cc:729] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.497498 16350 tablet_bootstrap.cc:654] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.497484 16349 consensus_queue.cc:260] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.497819 16349 ts_tablet_manager.cc:1398] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:33.499392 16348 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:33.499478 16348 ts_tablet_manager.cc:1367] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:57:33.499862 16348 raft_consensus.cc:348] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.499946 16348 raft_consensus.cc:374] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.499966 16348 raft_consensus.cc:729] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.501761 16344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "72d8e36923de4d1e8a6de93e73c84658" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:33.501897 16344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:33.500084 16348 consensus_queue.cc:260] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.507357 16348 ts_tablet_manager.cc:1398] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:33.508899 16286 raft_consensus.cc:482] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:33.509014 16286 raft_consensus.cc:504] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.509099 16133 catalog_manager.cc:2145] created HMS catalog entry for table [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:33.509608 16286 leader_election.cc:288] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:33.510025 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:33.510519 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:33.510663 15462 raft_consensus.cc:2454] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
W20240501 19:57:33.511211 15533 leader_election.cc:341] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:33.511401 15536 leader_election.cc:302] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters: 0c8dfc11dc7542f583644b9ac747545d
I20240501 19:57:33.512984 16286 raft_consensus.cc:2781] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:33.513051 16286 raft_consensus.cc:482] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:33.513089 16286 raft_consensus.cc:3037] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.514297 16133 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:37088 (request call id 11) took 1070 ms. Trace:
I20240501 19:57:33.514264 16286 raft_consensus.cc:504] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.514362 16133 rpcz_store.cc:276] 0501 19:57:32.444190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:32.449063 (+ 4873us) service_pool.cc:225] Handling call
0501 19:57:32.624055 (+174992us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:32.628437 (+ 4382us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:33.509090 (+880653us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:33.514049 (+ 4959us) catalog_manager.cc:2178] Wrote table and tablets to system table
0501 19:57:33.514183 (+ 134us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0501 19:57:33.514277 (+ 94us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:33.509632 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0501 19:57:33.509730 (+ 98us) write_op.cc:424] Acquiring schema lock in shared mode
0501 19:57:33.509734 (+ 4us) write_op.cc:427] Acquired schema lock
0501 19:57:33.509736 (+ 2us) tablet.cc:660] Decoding operations
0501 19:57:33.509802 (+ 66us) write_op.cc:612] Acquiring the partition lock for write op
0501 19:57:33.509813 (+ 11us) write_op.cc:633] Partition lock acquired for write op
0501 19:57:33.509816 (+ 3us) tablet.cc:683] Acquiring locks for 2 operations
0501 19:57:33.509874 (+ 58us) tablet.cc:699] Row locks acquired
0501 19:57:33.509875 (+ 1us) write_op.cc:260] PREPARE: finished
0501 19:57:33.509903 (+ 28us) write_op.cc:270] Start()
0501 19:57:33.509928 (+ 25us) write_op.cc:276] Timestamp: P: 1714593453509858 usec, L: 0
0501 19:57:33.509930 (+ 2us) op_driver.cc:331] REPLICATION: starting
0501 19:57:33.510067 (+ 137us) log.cc:842] Serialized 433 byte log entry
0501 19:57:33.513232 (+ 3165us) op_driver.cc:444] REPLICATION: finished
0501 19:57:33.513347 (+ 115us) write_op.cc:301] APPLY: starting
0501 19:57:33.513372 (+ 25us) tablet.cc:1365] starting BulkCheckPresence
0501 19:57:33.513407 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0501 19:57:33.513422 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0501 19:57:33.513516 (+ 94us) tablet.cc:1381] finished ApplyRowOperation cycle
0501 19:57:33.513548 (+ 32us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0501 19:57:33.513553 (+ 5us) write_op.cc:312] APPLY: finished
0501 19:57:33.513614 (+ 61us) log.cc:842] Serialized 29 byte log entry
0501 19:57:33.513881 (+ 267us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.513904 (+ 23us) write_op.cc:446] Released schema lock
0501 19:57:33.513911 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0501 19:57:33.513964 (+ 53us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.513966 (+ 2us) write_op.cc:446] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":842559,"Hive Metastore.run_cpu_time_us":230,"Hive Metastore.run_wall_time_us":37882,"spinlock_wait_cycles":896,"child_traces":[["op",{"apply.queue_time_us":76,"apply.run_cpu_time_us":462,"apply.run_wall_time_us":656,"num_ops":2,"prepare.queue_time_us":44,"prepare.run_cpu_time_us":519,"prepare.run_wall_time_us":518,"raft.queue_time_us":581,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":196,"replication_time_us":3277}]]}
I20240501 19:57:33.514621 16286 leader_election.cc:288] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:33.515558 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
W20240501 19:57:33.516305 15533 leader_election.cc:341] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:57:33.519627 16350 tablet_bootstrap.cc:492] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:33.519747 16350 ts_tablet_manager.cc:1367] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20240501 19:57:33.520355 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:33.520323 16350 raft_consensus.cc:348] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.520457 16350 raft_consensus.cc:374] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.520486 15462 raft_consensus.cc:3037] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.520504 16350 raft_consensus.cc:729] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.521302 16199 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:33.521462 16199 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
F20240501 19:57:33.521523 16199 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
I20240501 19:57:33.521618 15462 raft_consensus.cc:2454] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
*** Check failure stack trace: ***
I20240501 19:57:33.522037 15536 leader_election.cc:302] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters: 0c8dfc11dc7542f583644b9ac747545d
I20240501 19:57:33.522153 16350 consensus_queue.cc:260] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.524801 16350 ts_tablet_manager.cc:1398] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
@ 0x7f0897e7fdcd google::LogMessage::Fail() at ??:0
@ 0x7f0897e83b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f0897e7f7cc google::LogMessage::Flush() at ??:0
I20240501 19:57:33.530288 15303 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.530503 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:33.530637 15303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.530787 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.530905 16286 raft_consensus.cc:2781] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:33.530992 16286 raft_consensus.cc:686] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:33.531136 16286 consensus_queue.cc:237] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
@ 0x7f0897e7fa11 google::LogMessage::~LogMessage() at ??:0
I20240501 19:57:33.533293 16350 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:33.534489 16350 tablet_bootstrap.cc:654] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.535349 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f089cdab4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240501 19:57:33.535575 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.537308 16348 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:33.538255 16348 tablet_bootstrap.cc:654] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:33.539733 16349 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:33.540773 16349 tablet_bootstrap.cc:654] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
@ 0x7f089cdaa5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240501 19:57:33.546958 16133 catalog_manager.cc:5484] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f011d422e164c949a9d54d846b7c432 (127.14.96.3). New cstate: current_term: 1 leader_uuid: "7f011d422e164c949a9d54d846b7c432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } health_report { overall_health: HEALTHY } } }
@ 0x7f089cda9d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f089cdaefc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x560d75ac775c std::function<>::operator()() at ??:0
@ 0x7f08992c0807 kudu::Thread::SuperviseThread() at ??:0
I20240501 19:57:33.554421 16349 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:33.554565 16349 ts_tablet_manager.cc:1367] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:57:33.555151 16349 raft_consensus.cc:348] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.555290 16349 raft_consensus.cc:374] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.555323 16349 raft_consensus.cc:729] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.555450 16349 consensus_queue.cc:260] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.555756 16349 ts_tablet_manager.cc:1398] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
@ 0x7f0899afe6db start_thread at ??:0
I20240501 19:57:33.557822 15801 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=4315c5cbd36245a785ac6e19bb8806b3]
I20240501 19:57:33.558053 15801 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 39) took 3763 ms. Trace:
I20240501 19:57:33.558111 15801 rpcz_store.cc:276] 0501 19:57:29.794953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:31.186843 (+1391890us) service_pool.cc:225] Handling call
0501 19:57:32.425190 (+1238347us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:32.425531 (+ 341us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:33.345199 (+919668us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:33.345886 (+ 687us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.345888 (+ 2us) write_op.cc:446] Released schema lock
0501 19:57:33.346013 (+ 125us) catalog_manager.cc:2152] Rolling back HMS table creation
0501 19:57:33.558034 (+212021us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:33.345837 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER)
0501 19:57:33.345845 (+ 8us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.345850 (+ 5us) write_op.cc:446] Released schema lock
0501 19:57:33.345853 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":954440,"Hive Metastore.run_cpu_time_us":413,"Hive Metastore.run_wall_time_us":176521,"child_traces":[["op",{}]]}
W20240501 19:57:33.559101 15654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.61:46321)
@ 0x7f089703671f clone at ??:0
I20240501 19:57:33.564460 16348 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:33.564586 16348 ts_tablet_manager.cc:1367] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20240501 19:57:33.565132 16348 raft_consensus.cc:348] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.565281 16348 raft_consensus.cc:374] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.565316 16348 raft_consensus.cc:729] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.565461 16348 consensus_queue.cc:260] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.565793 16348 ts_tablet_manager.cc:1398] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:33.566403 16350 tablet_bootstrap.cc:492] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:33.566505 16350 ts_tablet_manager.cc:1367] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.001s
I20240501 19:57:33.567061 16350 raft_consensus.cc:348] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.567178 16350 raft_consensus.cc:374] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:33.567214 16350 raft_consensus.cc:729] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:33.567332 16350 consensus_queue.cc:260] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.567634 16350 ts_tablet_manager.cc:1398] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:33.577328 16276 raft_consensus.cc:482] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:33.577461 16276 raft_consensus.cc:504] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.577875 16276 leader_election.cc:288] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:33.578269 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a6e167a0d4040aac09233e544c73d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:33.578436 15326 raft_consensus.cc:2454] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:33.578780 15397 leader_election.cc:302] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:33.578955 16276 raft_consensus.cc:2781] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:33.579013 16276 raft_consensus.cc:482] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:33.579047 16276 raft_consensus.cc:3037] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.579329 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a6e167a0d4040aac09233e544c73d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:33.579478 15599 raft_consensus.cc:2454] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:33.580096 16276 raft_consensus.cc:504] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.580475 16276 leader_election.cc:288] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:33.580806 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a6e167a0d4040aac09233e544c73d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:33.580981 15326 raft_consensus.cc:3037] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.582010 15326 raft_consensus.cc:2454] T 568a6e167a0d4040aac09233e544c73d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:33.582360 15397 leader_election.cc:302] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:33.582540 16276 raft_consensus.cc:2781] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:33.582623 16276 raft_consensus.cc:686] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:33.582757 16276 consensus_queue.cc:237] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.582872 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "568a6e167a0d4040aac09233e544c73d" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:33.583002 15599 raft_consensus.cc:3037] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.584057 15599 raft_consensus.cc:2454] T 568a6e167a0d4040aac09233e544c73d P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:33.587719 16340 raft_consensus.cc:482] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:33.587829 16340 raft_consensus.cc:504] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.588289 16340 leader_election.cc:288] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:33.588827 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:33.589530 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be935df388a842d396a805bfc892496d" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:33.589685 15462 raft_consensus.cc:2379] T be935df388a842d396a805bfc892496d P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c8dfc11dc7542f583644b9ac747545d in current term 1: Already voted for candidate 7f011d422e164c949a9d54d846b7c432 in this term.
I20240501 19:57:33.590004 15262 leader_election.cc:302] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d; no voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432
I20240501 19:57:33.590265 16340 raft_consensus.cc:3037] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.591430 16340 raft_consensus.cc:2726] T be935df388a842d396a805bfc892496d P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:57:33.634814 16367 raft_consensus.cc:482] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:33.634956 16367 raft_consensus.cc:504] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.635489 16367 leader_election.cc:288] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:33.635808 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41bcd8b98a45a3b384f1642ae4396e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:33.635896 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41bcd8b98a45a3b384f1642ae4396e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:33.635946 15462 raft_consensus.cc:2454] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:33.636024 15326 raft_consensus.cc:2454] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:33.636289 15536 leader_election.cc:302] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:33.636451 16367 raft_consensus.cc:2781] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:33.636503 16367 raft_consensus.cc:482] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:33.636550 16367 raft_consensus.cc:3037] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.637924 16367 raft_consensus.cc:504] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.638271 16367 leader_election.cc:288] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:33.638972 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41bcd8b98a45a3b384f1642ae4396e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:33.639103 15462 raft_consensus.cc:3037] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.640161 15462 raft_consensus.cc:2454] T db41bcd8b98a45a3b384f1642ae4396e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:33.640573 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db41bcd8b98a45a3b384f1642ae4396e" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:33.640733 15326 raft_consensus.cc:3037] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.641716 15326 raft_consensus.cc:2454] T db41bcd8b98a45a3b384f1642ae4396e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:33.642220 15536 leader_election.cc:302] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:33.642885 16367 raft_consensus.cc:2781] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:33.642993 16367 raft_consensus.cc:686] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:33.643146 16367 consensus_queue.cc:237] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:33.647856 15803 catalog_manager.cc:2155] deleted orphaned HMS catalog entry for table [id=fa3e8422ce5a432fa46773338cc009e3]
I20240501 19:57:33.648085 15803 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:40740 (request call id 40) took 3853 ms. Trace:
I20240501 19:57:33.648144 15803 rpcz_store.cc:276] 0501 19:57:29.794986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 19:57:31.234992 (+1440006us) service_pool.cc:225] Handling call
0501 19:57:32.426006 (+1191014us) catalog_manager.cc:2072] Acquired catalog manager lock
0501 19:57:32.426299 (+ 293us) catalog_manager.cc:2118] Created new table and tablet info
0501 19:57:33.416688 (+990389us) catalog_manager.cc:2144] Created new table in HMS catalog
0501 19:57:33.417185 (+ 497us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.417186 (+ 1us) write_op.cc:446] Released schema lock
0501 19:57:33.417300 (+ 114us) catalog_manager.cc:2152] Rolling back HMS table creation
0501 19:57:33.648069 (+230769us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0501 19:57:33.417143 (+ 0us) op_driver.cc:372] HandleFailure(Illegal state: replica 62106a6dbb3846df9d3b29ab99fb3763 is not leader of this config: current role FOLLOWER)
0501 19:57:33.417150 (+ 7us) write_op.cc:481] Releasing partition, row and schema locks
0501 19:57:33.417154 (+ 4us) write_op.cc:446] Released schema lock
0501 19:57:33.417156 (+ 2us) write_op.cc:335] FINISH: Op aborted
Metrics: {"Hive Metastore.queue_time_us":1059016,"Hive Metastore.run_cpu_time_us":365,"Hive Metastore.run_wall_time_us":161712,"child_traces":[["op",{}]]}
I20240501 19:57:33.755635 16340 raft_consensus.cc:482] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:33.755787 16340 raft_consensus.cc:504] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.756229 16340 leader_election.cc:288] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:33.756757 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a3b7c06ec4067bc4fb9e8f91ffad3" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:33.756784 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a3b7c06ec4067bc4fb9e8f91ffad3" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:33.756938 15599 raft_consensus.cc:2454] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:33.757012 15462 raft_consensus.cc:2454] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:33.757306 15261 leader_election.cc:302] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:33.757501 16340 raft_consensus.cc:2781] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:33.757563 16340 raft_consensus.cc:482] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:33.757617 16340 raft_consensus.cc:3037] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.758476 16340 raft_consensus.cc:504] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:33.758818 16340 leader_election.cc:288] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:33.759238 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a3b7c06ec4067bc4fb9e8f91ffad3" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:33.759372 15599 raft_consensus.cc:3037] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.759388 15462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d2a3b7c06ec4067bc4fb9e8f91ffad3" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:33.759527 15462 raft_consensus.cc:3037] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:33.760191 15599 raft_consensus.cc:2454] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:33.760479 15261 leader_election.cc:302] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:33.760553 15462 raft_consensus.cc:2454] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:33.760655 16340 raft_consensus.cc:2781] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:33.760758 16340 raft_consensus.cc:686] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:33.760939 16340 consensus_queue.cc:237] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
W20240501 19:57:33.799845 15653 connection.cc:535] client connection to 127.14.96.61:46321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:33.799933 15533 connection.cc:535] client connection to 127.14.96.61:46321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:33.800405 15653 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.61:46321)
W20240501 19:57:33.800521 15653 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.61:46321)
W20240501 19:57:33.800654 15653 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.61:46321)
W20240501 19:57:33.802332 16324 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:33.801045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.14.96.61:46321 (local address 127.14.96.3:43147)
0501 19:57:33.801645 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0501 19:57:33.801702 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":494}
I20240501 19:57:33.938234 16286 consensus_queue.cc:1035] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.942188 16286 consensus_queue.cc:1035] T be935df388a842d396a805bfc892496d P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:33.942798 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--webserver_interface=127.14.96.62
--webserver_port=42813
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:34.052798 16394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:34.053078 16394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.054390 16394 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:34.054317 16391 consensus_queue.cc:1035] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:57:34.055272 15789 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:34.055446 15787 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:34.055514 15787 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:34.056456 16394 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 19:57:34.059774 16367 consensus_queue.cc:1035] T db41bcd8b98a45a3b384f1642ae4396e P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:57:34.061193 16393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:34.061563 16393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:34.061672 16393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:34.065429 16393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:34.065490 16393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:34.065511 16393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:34.065531 16393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:34.069581 16393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.96.62
--webserver_port=42813
--never_fsync=true
--heap_profile_path=/tmp/kudu.16393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:34.070623 16393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:34.071877 16393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:34.080307 16405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:34.080858 16404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:34.081996 16407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:34.082738 16393 server_base.cc:1034] running on GCE node
I20240501 19:57:34.083266 16393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:34.084264 16393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:34.085428 16393 hybrid_clock.cc:648] HybridClock initialized: now 1714593454085403 us; error 40 us; skew 500 ppm
I20240501 19:57:34.087508 16393 webserver.cc:460] Webserver started at http://127.14.96.62:42813/ using document root <none> and password file <none>
I20240501 19:57:34.088127 16393 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:34.088201 16393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:34.091889 16393 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:34.094022 16413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:34.095099 16393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:34.095217 16393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:34.095644 16393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:34.120365 15741 consensus_queue.cc:1035] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:34.122094 16393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:34.122951 16393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:34.123183 16393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:34.127372 16276 consensus_queue.cc:1035] T 568a6e167a0d4040aac09233e544c73d P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:34.131968 16393 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.62:37747
I20240501 19:57:34.133237 16393 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:34.131968 16468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.62:37747 every 8 connection(s)
I20240501 19:57:34.137883 16469 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:34.139768 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap starting.
I20240501 19:57:34.141111 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16393
I20240501 19:57:34.155105 16469 log.cc:824] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:34.207753 16311 consensus_queue.cc:1035] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:34.214916 16340 consensus_queue.cc:1035] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:34.220399 15647 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:34.222379 15373 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:34.224959 15511 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:34.230851 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=140 overwritten=0 applied=140 ignored=0} inserts{seen=95 ignored=0} mutations{seen=123 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:34.231254 16469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap complete.
I20240501 19:57:34.234225 16469 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.234653 16469 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Initialized, Role: FOLLOWER
I20240501 19:57:34.235152 16469 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 140, Last appended: 3.140, Last appended by leader: 140, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.235628 16478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:34.235738 16478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:34.236372 16469 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:34.243311 16492 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:34.243408 16492 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: loading cluster ID for follower catalog manager: success
I20240501 19:57:34.247051 16492 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: acquiring CA information for follower catalog manager: success
I20240501 19:57:34.253345 16492 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:34.257261 16393 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:34.259596 16393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:34.446774 15720 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:34.446905 15720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:34.468438 16428 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:57211; Asking this server to re-register.
I20240501 19:57:34.468976 15647 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:34.470337 16428 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:34.639042 16428 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:55647; Asking this server to re-register.
I20240501 19:57:34.639456 15511 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:34.640424 16428 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:34.693786 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:34.694000 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.695256 16478 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:34.696122 16414 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:34.696905 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:34.698853 15807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 140 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:34.699043 15807 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9f177158f524f0183f6eec721ea83e4 for term 4 because replica has last-logged OpId of term: 3 index: 151, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 140.
I20240501 19:57:34.699417 16415 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:34.699666 16478 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240501 19:57:34.726087 16428 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:46019; Asking this server to re-register.
I20240501 19:57:34.726606 15373 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:34.727615 16428 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:34.778928 16503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:34.779062 16503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.779528 16503 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:34.780033 15787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20240501 19:57:34.780079 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 4 candidate_status { last_received { term: 3 index: 151 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
I20240501 19:57:34.780426 16437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 3.
I20240501 19:57:34.780786 15789 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters:
W20240501 19:57:34.780929 15787 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:34.780992 16503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240501 19:57:34.781046 16503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 72d8e36923de4d1e8a6de93e73c84658)
I20240501 19:57:34.781090 16503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 3 FOLLOWER]: Advancing to term 4
I20240501 19:57:34.781939 16503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.782222 16503 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:34.782603 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 4 candidate_status { last_received { term: 3 index: 151 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:34.782732 16437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 3 FOLLOWER]: Advancing to term 4
W20240501 19:57:34.783133 15787 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:34.784617 16437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 4.
I20240501 19:57:34.784933 15789 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:34.785086 16503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 19:57:34.785264 16503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 LEADER]: Becoming Leader. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Running, Role: LEADER
I20240501 19:57:34.785404 16503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 3.151, Last appended by leader: 151, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:34.786032 16505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 4 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:34.786161 16505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:34.786468 16507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:34.788251 16507 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:34.788784 16507 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:34.789008 16507 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:34.789202 16507 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:34.789402 16507 catalog_manager.cc:669] Loaded metadata for table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:34.789592 16507 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:34.789777 16507 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:34.789975 16507 catalog_manager.cc:669] Loaded metadata for table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:34.790180 16507 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:34.790455 16507 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:34.790632 16507 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:34.790840 16507 catalog_manager.cc:669] Loaded metadata for table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:34.791019 16507 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:34.791294 16507 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:34.791524 16507 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:34.791713 16507 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:34.791977 16507 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:34.792160 16507 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:34.792356 16507 catalog_manager.cc:669] Loaded metadata for table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:34.794338 16507 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:34.794683 16507 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:34.795024 16507 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:34.795271 16507 tablet_loader.cc:96] loaded metadata for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:34.795514 16507 tablet_loader.cc:96] loaded metadata for tablet 568a6e167a0d4040aac09233e544c73d (table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1])
I20240501 19:57:34.795667 16507 tablet_loader.cc:96] loaded metadata for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71])
I20240501 19:57:34.795823 16507 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:34.796226 16507 tablet_loader.cc:96] loaded metadata for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:34.796460 16507 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:34.796613 16507 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:34.797000 16507 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:34.797262 16507 tablet_loader.cc:96] loaded metadata for tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:34.797418 16507 tablet_loader.cc:96] loaded metadata for tablet be935df388a842d396a805bfc892496d (table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0])
I20240501 19:57:34.797571 16507 tablet_loader.cc:96] loaded metadata for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:34.797806 16507 tablet_loader.cc:96] loaded metadata for tablet db41bcd8b98a45a3b384f1642ae4396e (table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af])
I20240501 19:57:34.798036 16507 tablet_loader.cc:96] loaded metadata for tablet e57997f8a3a742ed86ca9dca11161071 (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:34.798353 16507 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:34.798686 16507 tablet_loader.cc:96] loaded metadata for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:34.798840 16507 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:34.799041 16507 tablet_loader.cc:96] loaded metadata for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:34.799401 16507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:34.800412 16507 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:34.800462 16507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:34.801560 16507 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:34.802433 16507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Loaded TSK: 0
I20240501 19:57:34.802699 16507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:34.803519 16507 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:34.804497 16507 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 45
I20240501 19:57:34.809581 15803 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_7a325fdcd5224571b886231cf98de803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.809646 15801 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_a283fbed2b5748f1b138b73c95bb2dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.809942 15797 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_06ea968fee694fad8d382f41b03ea362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.809999 15800 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_3fd15f4a5d9f4fe2a7062c4fb1c53392" } modify_external_catalogs: true
I20240501 19:57:34.810037 15798 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_f18a22bd8291471ba13555510194e198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.810115 15800 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_3fd15f4a5d9f4fe2a7062c4fb1c53392" } modify_external_catalogs: true
I20240501 19:57:34.810165 15796 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:40740:
table { table_name: "default.Table_b16d0f8f6d3341c8bbd83fc19a8c3771" } new_table_name: "default.Table_ffecde6d8bcf4ef58ab7d34e7904449b" modify_external_catalogs: true
I20240501 19:57:34.810660 15802 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_206463f213f945b2a74cbbcddbada5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.810870 15799 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_abc8d72a6cc74062a62861c18c185a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.811070 15805 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_84bf2da3cdaf45fa84c6099c013783e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.811281 15804 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40740:
name: "default.Table_a62bb6dcb7644aa4bb9008b9f9a196dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:34.814771 15859 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:34.814954 15859 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
F20240501 19:57:34.815016 15859 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
*** Check failure stack trace: ***
@ 0x7f938aad0dcd google::LogMessage::Fail() at ??:0
@ 0x7f938aad4b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f938aad07cc google::LogMessage::Flush() at ??:0
@ 0x7f938aad0a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f938f9fc4f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f938f9fb5ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f938f9fad51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f938f9fffc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55f24105275c std::function<>::operator()() at ??:0
@ 0x7f938bf11807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f938c74f6db start_thread at ??:0
@ 0x7f9389c8771f clone at ??:0
W20240501 19:57:34.999477 15654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:34.999657 15654 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:34.999799 15654 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:35.000175 15654 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.60:40075)
I20240501 19:57:35.718654 16524 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:35.718796 16524 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:35.719257 16524 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:35.720281 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:35.720414 16415 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:35.720494 16415 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:35.720638 16524 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 19:57:36.653021 16537 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:36.653149 16537 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:36.653532 16537 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:36.654670 16415 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:36.654753 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:36.654825 16414 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:36.654942 16537 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 19:57:37.161537 16547 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:37.161669 16547 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:37.162273 16547 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:37.163198 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:37.163296 16415 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:37.163378 16415 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:37.163511 16547 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240501 19:57:37.256325 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--webserver_interface=127.14.96.60
--webserver_port=43311
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:37.372884 16553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:37.373137 16553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:37.373176 16553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:37.376364 16553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:37.376447 16553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:37.376468 16553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:37.376490 16553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:37.380885 16553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.96.60
--webserver_port=43311
--never_fsync=true
--heap_profile_path=/tmp/kudu.16553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:37.381883 16553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:37.383124 16553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:37.391073 16558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:37.391690 16561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:37.392293 16559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:37.393194 16553 server_base.cc:1034] running on GCE node
I20240501 19:57:37.393744 16553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:37.394726 16553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:37.395876 16553 hybrid_clock.cc:648] HybridClock initialized: now 1714593457395855 us; error 42 us; skew 500 ppm
I20240501 19:57:37.403638 16553 webserver.cc:460] Webserver started at http://127.14.96.60:43311/ using document root <none> and password file <none>
I20240501 19:57:37.404290 16553 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:37.404363 16553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:37.408231 16553 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:37.410475 16567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:37.411521 16553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:37.411636 16553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:37.412063 16553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:37.432142 16553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:37.432950 16553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:37.433127 16553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:37.440258 16553 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.60:40075
I20240501 19:57:37.440259 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.60:40075 every 8 connection(s)
I20240501 19:57:37.441572 16553 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
I20240501 19:57:37.443331 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16553
I20240501 19:57:37.446537 16620 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:37.448901 16620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap starting.
I20240501 19:57:37.462105 16620 log.cc:824] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:37.532064 16620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap replayed 1/1 log segments. Stats: ops{read=152 overwritten=0 applied=151 ignored=0} inserts{seen=103 ignored=0} mutations{seen=131 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:37.532572 16620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap complete.
I20240501 19:57:37.535956 16620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:37.536522 16620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Initialized, Role: FOLLOWER
I20240501 19:57:37.537007 16620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 4.152, Last appended by leader: 152, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:37.537536 16625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:37.537676 16625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:37.538228 16620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:37.547488 16639 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:37.547572 16639 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: loading cluster ID for follower catalog manager: success
I20240501 19:57:37.551119 16639 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: acquiring CA information for follower catalog manager: success
I20240501 19:57:37.554426 16639 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:37.553035 16553 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:37.555503 16553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:37.629824 15646 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:37.632226 16581 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:35181; Asking this server to re-register.
I20240501 19:57:37.632824 15646 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:37.634404 16581 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:37.648713 15510 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:37.650189 16581 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:38095; Asking this server to re-register.
I20240501 19:57:37.650717 15510 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:37.651784 16581 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:37.756914 15372 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:37.758142 16581 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:47941; Asking this server to re-register.
I20240501 19:57:37.758639 15372 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:37.759788 16581 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:38.053966 16650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:38.054180 16650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:38.055410 16650 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:38.055919 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 5 candidate_status { last_received { term: 4 index: 152 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4" is_pre_election: true
W20240501 19:57:38.056049 16568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:38.056092 16437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 4.
I20240501 19:57:38.056480 16570 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters:
I20240501 19:57:38.056735 16650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240501 19:57:38.056800 16650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:38.056874 16650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 4 FOLLOWER]: Advancing to term 5
W20240501 19:57:38.056864 16568 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:38.058869 16650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:38.059260 16650 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:38.059592 16437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 5 candidate_status { last_received { term: 4 index: 152 } } ignore_live_leader: false dest_uuid: "e9f177158f524f0183f6eec721ea83e4"
I20240501 19:57:38.059712 16437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 4 FOLLOWER]: Advancing to term 5
W20240501 19:57:38.060254 16568 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:38.060858 16437 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 5.
I20240501 19:57:38.061264 16570 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:38.061432 16650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 19:57:38.061580 16650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 LEADER]: Becoming Leader. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Running, Role: LEADER
I20240501 19:57:38.061864 16650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 151, Committed index: 151, Last appended: 4.152, Last appended by leader: 152, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:38.063472 16653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.063597 16653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:38.063885 16655 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:38.066008 16655 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:38.066619 16655 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:38.066831 16655 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:38.067034 16655 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:38.067240 16655 catalog_manager.cc:669] Loaded metadata for table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:38.067482 16655 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:38.067689 16655 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:38.067900 16655 catalog_manager.cc:669] Loaded metadata for table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:38.068114 16655 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:38.068390 16655 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:38.068578 16655 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:38.068765 16655 catalog_manager.cc:669] Loaded metadata for table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:38.068980 16655 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:38.069274 16655 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:38.069468 16655 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:38.069658 16655 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:38.069931 16655 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:38.070125 16655 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:38.070317 16655 catalog_manager.cc:669] Loaded metadata for table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:38.072571 16655 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:38.073047 16655 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:38.073483 16655 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:38.073747 16655 tablet_loader.cc:96] loaded metadata for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:38.073994 16655 tablet_loader.cc:96] loaded metadata for tablet 568a6e167a0d4040aac09233e544c73d (table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1])
I20240501 19:57:38.074156 16655 tablet_loader.cc:96] loaded metadata for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71])
I20240501 19:57:38.074323 16655 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:38.074734 16655 tablet_loader.cc:96] loaded metadata for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:38.074975 16655 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:38.075162 16655 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:38.075582 16655 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:38.075845 16655 tablet_loader.cc:96] loaded metadata for tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:38.076004 16655 tablet_loader.cc:96] loaded metadata for tablet be935df388a842d396a805bfc892496d (table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0])
I20240501 19:57:38.076164 16655 tablet_loader.cc:96] loaded metadata for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:38.076421 16655 tablet_loader.cc:96] loaded metadata for tablet db41bcd8b98a45a3b384f1642ae4396e (table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af])
I20240501 19:57:38.076673 16655 tablet_loader.cc:96] loaded metadata for tablet e57997f8a3a742ed86ca9dca11161071 (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:38.077045 16655 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:38.077381 16655 tablet_loader.cc:96] loaded metadata for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:38.077543 16655 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:38.077706 16655 tablet_loader.cc:96] loaded metadata for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:38.077864 16655 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:38.078855 16655 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:38.078908 16655 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:38.080008 16655 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:38.081102 16655 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Loaded TSK: 0
I20240501 19:57:38.081928 16655 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:38.082917 16655 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:38.083760 16655 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 45
I20240501 19:57:38.125625 16584 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:38.125699 16581 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33706:
name: "default.Table_a82d93e7c3e949e99304ee3a61f2f5e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:38.125849 16584 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:38.126199 16578 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_8d1020ff0b064f7199ad44d673369f79" } new_table_name: "default.Table_5dc76a7d16884cc59719736ac8692923" modify_external_catalogs: true
I20240501 19:57:38.126291 16583 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:38.126334 16576 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:38.126480 16582 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5" } new_table_name: "default.Table_557eeb62e4e64cc6aeca0b1882dd3909" modify_external_catalogs: true
I20240501 19:57:38.126514 16577 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_090e62cd0c6c43eb9c44ada659e01c98" } new_table_name: "default.Table_f0938e2fdba84a2ca4f93e39ab3e7eac" modify_external_catalogs: true
I20240501 19:57:38.126724 16580 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:38.126767 16579 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_b7b25ac22dba40c9aa0a509437741d50" } new_table_name: "default.Table_3846033cb97a4827a54a0f1bcf875141" modify_external_catalogs: true
I20240501 19:57:38.126819 16580 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:38.126363 16583 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:38.126919 16575 master_service.cc:946] ReplaceTablet: received request to replace tablet e57997f8a3a742ed86ca9dca11161071 from {username='slave'} at 127.0.0.1:33706
I20240501 19:57:38.126487 16576 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33706:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:38.132354 16638 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:38.132696 16638 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
F20240501 19:57:38.132755 16638 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
*** Check failure stack trace: ***
@ 0x7fe444898dcd google::LogMessage::Fail() at ??:0
I20240501 19:57:38.136435 16437 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Refusing update from remote peer 62106a6dbb3846df9d3b29ab99fb3763: Log matching property violated. Preceding OpId in replica: term: 3 index: 140. Preceding OpId from leader: term: 5 index: 154. (index mismatch)
W20240501 19:57:38.138098 16568 consensus_peers.cc:485] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 -> Peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Couldn't send request to peer 72d8e36923de4d1e8a6de93e73c84658. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fe44489cb93 google::LogMessage::SendToLog() at ??:0
I20240501 19:57:38.138715 16653 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 153, Last known committed idx: 140, Time since last communication: 0.000s
@ 0x7fe4448987cc google::LogMessage::Flush() at ??:0
@ 0x7fe444898a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fe4497c44f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7fe4497c35ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240501 19:57:38.151419 16650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.151599 16650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:38.151736 16656 mvcc.cc:205] Tried to move back new op lower bound from 7022974804497195008 to 7022974790802329600. Current Snapshot: MvccSnapshot[applied={T|T < 7022974804497195008}]
I20240501 19:57:38.151899 16659 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.152010 16659 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:38.152400 16650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.152529 16650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
@ 0x7fe4497c2d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240501 19:57:38.154887 16575 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet e57997f8a3a742ed86ca9dca11161071
I20240501 19:57:38.155486 16661 mvcc.cc:205] Tried to move back new op lower bound from 7022974804497195008 to 7022974790802329600. Current Snapshot: MvccSnapshot[applied={T|T < 7022974785615888384}]
I20240501 19:57:38.156280 16575 catalog_manager.cc:6554] ReplaceTablet: tablet e57997f8a3a742ed86ca9dca11161071 deleted and replaced by tablet 956a62d864b14bf7b433c7fd7c335d1e
I20240501 19:57:38.156641 16671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.156754 16671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:38.156827 16671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:38.156910 16671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7fe4497c7fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x564b7816575c std::function<>::operator()() at ??:0
@ 0x7fe445cd9807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4465176db start_thread at ??:0
@ 0x7fe443a4f71f clone at ??:0
W20240501 19:57:38.328658 15260 connection.cc:535] client connection to 127.14.96.60:40075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:38.328889 16676 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:38.163537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38564 (local address 127.14.96.3:42023)
0501 19:57:38.166171 (+ 2634us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:38.166176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:38.328713 (+162537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38564: BlockingRecv error: recv got EOF from 127.0.0.1:38564 (error 108)
Metrics: {"server-negotiator.queue_time_us":2548,"thread_start_us":83,"threads_started":1}
W20240501 19:57:38.329319 16674 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:38.162311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.14.96.1:36247)
0501 19:57:38.165686 (+ 3375us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:38.165693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:38.165711 (+ 18us) server_negotiation.cc:407] Connection header received
0501 19:57:38.165769 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:38.165774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:38.165839 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:38.165927 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:38.328721 (+162794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37624: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":3255,"thread_start_us":106,"threads_started":1}
W20240501 19:57:38.329339 16672 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:57:38.157588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.14.96.2:34181)
0501 19:57:38.165657 (+ 8069us) server_negotiation.cc:206] Beginning negotiation
0501 19:57:38.165664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0501 19:57:38.165685 (+ 21us) server_negotiation.cc:407] Connection header received
0501 19:57:38.165747 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0501 19:57:38.165753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0501 19:57:38.165819 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0501 19:57:38.165968 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0501 19:57:38.328724 (+162756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44534: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":7952,"thread_start_us":81,"threads_started":1}
W20240501 19:57:38.329594 16414 connection.cc:535] server connection from 127.0.0.1:49848 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:38.330386 15654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:38.330951 15654 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:38.331063 15654 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:38.331610 15654 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.60:40075)
I20240501 19:57:38.665012 16689 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:38.665184 16689 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:38.665418 15646 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
I20240501 19:57:38.665629 16689 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:38.666941 16415 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:38.667064 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:38.667151 16414 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:38.667315 16689 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 19:57:39.261835 16393 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:39.261952 16393 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:39.305040 16697 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:39.305204 16697 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:39.305675 16697 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:39.306797 16414 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:39.306923 16415 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:39.306998 16415 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:39.307135 16697 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 19:57:39.913826 14720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu with pid 16393
I20240501 19:57:40.912393 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--webserver_interface=127.14.96.62
--webserver_port=42813
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:41.032157 16703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:41.032426 16703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:41.032469 16703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:41.035915 16703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:41.035995 16703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:41.036017 16703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:41.036036 16703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:41.040053 16703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.96.62
--webserver_port=42813
--never_fsync=true
--heap_profile_path=/tmp/kudu.16703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:41.041180 16703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:41.042635 16703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:41.051339 16708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:41.051923 16711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:41.052588 16709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:41.053287 16703 server_base.cc:1034] running on GCE node
I20240501 19:57:41.053853 16703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:41.054874 16703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:41.056027 16703 hybrid_clock.cc:648] HybridClock initialized: now 1714593461056016 us; error 28 us; skew 500 ppm
I20240501 19:57:41.058189 16703 webserver.cc:460] Webserver started at http://127.14.96.62:42813/ using document root <none> and password file <none>
I20240501 19:57:41.058835 16703 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:41.058910 16703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:41.062465 16703 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:41.064769 16717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:41.065857 16703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:57:41.065982 16703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:41.066432 16703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:41.086781 16703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:41.087509 16703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:41.087674 16703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:41.094909 16769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.62:37747 every 8 connection(s)
I20240501 19:57:41.094906 16703 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.62:37747
I20240501 19:57:41.096119 16703 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:41.098904 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16703
I20240501 19:57:41.100621 16770 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:41.102738 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap starting.
I20240501 19:57:41.116704 16770 log.cc:824] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:41.144268 15647 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:41.163735 15511 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:41.192571 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=155 overwritten=0 applied=154 ignored=0} inserts{seen=104 ignored=0} mutations{seen=132 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:41.193102 16770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap complete.
I20240501 19:57:41.196390 16770 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:41.197952 16770 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Initialized, Role: FOLLOWER
I20240501 19:57:41.198415 16770 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 154, Last appended: 5.155, Last appended by leader: 155, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:41.198807 16775 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:41.198923 16775 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:41.199479 16770 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:41.207437 16789 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:41.207599 16789 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: loading cluster ID for follower catalog manager: success
I20240501 19:57:41.212344 16789 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: acquiring CA information for follower catalog manager: success
I20240501 19:57:41.217501 16789 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:41.220717 16703 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:41.223001 16703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:41.270432 15373 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:41.272095 16734 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:42563; Asking this server to re-register.
I20240501 19:57:41.272641 15373 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:41.274140 16734 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:41.612411 16775 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:41.612655 16775 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:41.613925 16775 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:41.614462 16718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:41.615187 16718 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:41.615310 16719 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:41.615404 16719 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:41.615725 16775 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 19:57:41.647166 16734 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:38627; Asking this server to re-register.
I20240501 19:57:41.647614 15647 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:41.648686 16734 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:41.666452 16734 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:46583; Asking this server to re-register.
I20240501 19:57:41.667033 15511 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:41.668176 16734 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:42.272955 16797 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:42.273090 16797 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:42.273559 16797 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:42.275014 16718 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:42.275257 16719 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:42.275399 16719 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:42.275559 16797 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 19:57:43.148870 16802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:43.149030 16802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:43.149521 16802 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:43.150846 16719 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:43.150956 16718 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:43.151028 16718 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f177158f524f0183f6eec721ea83e4; no voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:43.151170 16802 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240501 19:57:44.479697 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--webserver_interface=127.14.96.60
--webserver_port=43311
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:44.598016 16807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:44.598275 16807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:44.598320 16807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:44.601637 16807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:44.601708 16807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:44.601729 16807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:44.601747 16807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:44.605737 16807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.60:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.14.96.60
--webserver_port=43311
--never_fsync=true
--heap_profile_path=/tmp/kudu.16807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:44.606731 16807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:44.607889 16807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:44.615975 16815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:44.616473 16812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:44.617038 16813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:44.617741 16807 server_base.cc:1034] running on GCE node
I20240501 19:57:44.618309 16807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:44.619295 16807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:44.620460 16807 hybrid_clock.cc:648] HybridClock initialized: now 1714593464620441 us; error 46 us; skew 500 ppm
I20240501 19:57:44.627458 16807 webserver.cc:460] Webserver started at http://127.14.96.60:43311/ using document root <none> and password file <none>
I20240501 19:57:44.628105 16807 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:44.628180 16807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:44.631847 16807 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:44.634645 16821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:44.636047 16807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:44.636186 16807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
uuid: "62106a6dbb3846df9d3b29ab99fb3763"
format_stamp: "Formatted at 2024-05-01 19:57:25 on dist-test-slave-spj6"
I20240501 19:57:44.636655 16807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:44.652266 16807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:44.653023 16807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:44.653174 16807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:44.660694 16807 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.60:40075
I20240501 19:57:44.660727 16873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.60:40075 every 8 connection(s)
I20240501 19:57:44.661881 16807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-2/data/info.pb
I20240501 19:57:44.666270 16874 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:44.666414 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16807
I20240501 19:57:44.668401 16874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap starting.
I20240501 19:57:44.683943 16874 log.cc:824] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:44.684715 15646 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:44.684906 15646 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
I20240501 19:57:44.746794 16878 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:44.746951 16878 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:44.747411 16878 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:44.748651 16718 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:44.759990 16874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap replayed 1/1 log segments. Stats: ops{read=155 overwritten=0 applied=154 ignored=0} inserts{seen=104 ignored=0} mutations{seen=132 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:57:44.760558 16874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Bootstrap complete.
I20240501 19:57:44.764930 16874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:44.766474 16874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Initialized, Role: FOLLOWER
I20240501 19:57:44.767027 16874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 154, Last appended: 5.155, Last appended by leader: 155, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:44.767468 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:44.767602 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:44.768200 16874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:44.768258 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 155 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:44.768548 16848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f177158f524f0183f6eec721ea83e4 in term 5.
I20240501 19:57:44.768980 16719 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:44.769232 16878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240501 19:57:44.769322 16878 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:44.769381 16878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 19:57:44.771726 16878 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:44.772145 16878 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 election: Requested vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:44.772629 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 155 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763"
I20240501 19:57:44.772876 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 5 FOLLOWER]: Advancing to term 6
W20240501 19:57:44.773252 16718 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:44.775043 16848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f177158f524f0183f6eec721ea83e4 in term 6.
I20240501 19:57:44.775386 16719 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:44.776969 16878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 19:57:44.777218 16878 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 LEADER]: Becoming Leader. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Running, Role: LEADER
I20240501 19:57:44.777565 16878 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 154, Committed index: 154, Last appended: 5.155, Last appended by leader: 155, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:44.779021 16878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f177158f524f0183f6eec721ea83e4. Latest consensus state: current_term: 6 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:44.779147 16878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:44.783463 16897 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:44.783536 16897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: loading cluster ID for follower catalog manager: success
I20240501 19:57:44.788995 16807 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:44.791409 16807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:44.791369 16897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: acquiring CA information for follower catalog manager: success
I20240501 19:57:44.794457 16897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:44.846052 15510 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:44.846323 15372 heartbeater.cc:347] Connected to a master server at 127.14.96.60:40075
I20240501 19:57:44.847877 16837 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:57993; Asking this server to re-register.
I20240501 19:57:44.847877 16838 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:54305; Asking this server to re-register.
I20240501 19:57:44.848511 15372 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:44.848546 15510 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:44.850081 16837 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:44.850162 16838 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:45.187987 15646 heartbeater.cc:503] Master 127.14.96.60:40075 was elected leader, sending a full tablet report...
I20240501 19:57:45.189855 16837 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:52505; Asking this server to re-register.
I20240501 19:57:45.190402 15646 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:45.191558 16837 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:45.253136 16848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Refusing update from remote peer e9f177158f524f0183f6eec721ea83e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 155. Preceding OpId from leader: term: 6 index: 156. (index mismatch)
I20240501 19:57:45.253841 16902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 154, Time since last communication: 0.000s
I20240501 19:57:45.255751 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f177158f524f0183f6eec721ea83e4. Latest consensus state: current_term: 6 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:45.255892 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:45.257567 16878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:45.257707 16878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:45.257910 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:45.258018 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 19:57:45.258949 16718 consensus_peers.cc:485] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 -> Peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Couldn't send request to peer 72d8e36923de4d1e8a6de93e73c84658. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:45.306607 16904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:45.308902 16904 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:45.309541 16904 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:45.309765 16904 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:45.309981 16904 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:45.310199 16904 catalog_manager.cc:669] Loaded metadata for table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:45.310406 16904 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:45.310613 16904 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:45.310808 16904 catalog_manager.cc:669] Loaded metadata for table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:45.311014 16904 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:45.311296 16904 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:45.311496 16904 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:45.311695 16904 catalog_manager.cc:669] Loaded metadata for table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:45.311885 16904 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:45.312178 16904 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:45.312377 16904 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:45.312575 16904 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:45.312903 16904 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:45.313172 16904 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:45.313401 16904 catalog_manager.cc:669] Loaded metadata for table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:45.316541 16904 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:45.317082 16904 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:45.317476 16904 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:45.317734 16904 tablet_loader.cc:96] loaded metadata for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:45.317987 16904 tablet_loader.cc:96] loaded metadata for tablet 568a6e167a0d4040aac09233e544c73d (table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1])
I20240501 19:57:45.318166 16904 tablet_loader.cc:96] loaded metadata for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71])
I20240501 19:57:45.318333 16904 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:45.318787 16904 tablet_loader.cc:96] loaded metadata for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:45.319111 16904 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:45.319361 16904 tablet_loader.cc:96] loaded metadata for tablet 956a62d864b14bf7b433c7fd7c335d1e (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:45.319573 16904 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:45.320070 16904 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:45.320422 16904 tablet_loader.cc:96] loaded metadata for tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:45.320632 16904 tablet_loader.cc:96] loaded metadata for tablet be935df388a842d396a805bfc892496d (table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0])
I20240501 19:57:45.320828 16904 tablet_loader.cc:96] loaded metadata for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:45.321208 16904 tablet_loader.cc:96] loaded metadata for tablet db41bcd8b98a45a3b384f1642ae4396e (table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af])
I20240501 19:57:45.321909 16904 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:45.322440 16904 tablet_loader.cc:96] loaded metadata for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:45.322599 16904 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:45.322731 16904 tablet_loader.cc:96] loaded metadata for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:45.322875 16904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:45.324595 16904 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:45.324651 16904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:45.325783 16904 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:45.326637 16904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Loaded TSK: 0
I20240501 19:57:45.327370 16904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:45.328233 16904 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:45.329087 16904 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 45
I20240501 19:57:45.414824 16732 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_206463f213f945b2a74cbbcddbada5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.415131 16731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_abc8d72a6cc74062a62861c18c185a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.415378 16729 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_a62bb6dcb7644aa4bb9008b9f9a196dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.415154 16730 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_84bf2da3cdaf45fa84c6099c013783e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.414824 16733 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_a283fbed2b5748f1b138b73c95bb2dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.414824 16734 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_7a325fdcd5224571b886231cf98de803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.415812 16728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_06ea968fee694fad8d382f41b03ea362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.416272 16727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50042:
name: "default.Table_f18a22bd8291471ba13555510194e198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:45.416335 16726 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:50042:
table { table_name: "default.Table_b16d0f8f6d3341c8bbd83fc19a8c3771" } new_table_name: "default.Table_ffecde6d8bcf4ef58ab7d34e7904449b" modify_external_catalogs: true
I20240501 19:57:45.416366 16725 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50042:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:45.416510 16725 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50042:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:45.420768 16788 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:45.421097 16788 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
F20240501 19:57:45.421167 16788 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
*** Check failure stack trace: ***
@ 0x7f9e7be5ddcd google::LogMessage::Fail() at ??:0
@ 0x7f9e7be61b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f9e7be5d7cc google::LogMessage::Flush() at ??:0
@ 0x7f9e7be5da11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f9e80d894f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
@ 0x7f9e80d885ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
@ 0x7f9e80d87d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
@ 0x7f9e80d8cfc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55932314b75c std::function<>::operator()() at ??:0
@ 0x7f9e7d29e807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9e7dadc6db start_thread at ??:0
@ 0x7f9e7b01471f clone at ??:0
W20240501 19:57:45.605991 15655 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.62:37747)
W20240501 19:57:45.606185 15655 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.62:37747)
W20240501 19:57:45.606290 15655 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.62:37747)
W20240501 19:57:45.606684 15655 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.62:37747)
I20240501 19:57:45.757994 16883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:45.758143 16883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:45.759389 16883 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:45.759893 16824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:45.760563 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:45.760773 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:45.760865 16824 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:45.761070 16883 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 19:57:46.378191 16932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:46.378331 16932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:46.378760 16932 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:46.380017 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:46.380129 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:46.380255 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:46.380426 16932 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 19:57:46.905120 16938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:46.905268 16938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:46.905753 16938 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:46.906900 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:46.907156 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:46.907228 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:46.907361 16938 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240501 19:57:47.719074 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--webserver_interface=127.14.96.62
--webserver_port=42813
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:47.844313 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:47.844589 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:47.844648 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:47.848306 16942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:47.848384 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:47.848419 16942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:47.848461 16942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:47.852689 16942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.62:37747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.14.96.62
--webserver_port=42813
--never_fsync=true
--heap_profile_path=/tmp/kudu.16942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:47.853703 16942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:47.854857 16942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:47.863359 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:47.863873 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:47.864511 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:47.865257 16942 server_base.cc:1034] running on GCE node
I20240501 19:57:47.865886 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:47.866925 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:47.868122 16942 hybrid_clock.cc:648] HybridClock initialized: now 1714593467868102 us; error 34 us; skew 500 ppm
I20240501 19:57:47.870852 16942 webserver.cc:460] Webserver started at http://127.14.96.62:42813/ using document root <none> and password file <none>
I20240501 19:57:47.871640 16942 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:47.871718 16942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:47.875399 16942 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:47.877472 16956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:47.878530 16942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:47.878646 16942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
uuid: "e9f177158f524f0183f6eec721ea83e4"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:47.879123 16942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:47.894990 16942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:47.895794 16942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:47.895972 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:47.903295 16942 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.62:37747
I20240501 19:57:47.903309 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.62:37747 every 8 connection(s)
I20240501 19:57:47.904421 16942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-0/data/info.pb
I20240501 19:57:47.908706 17009 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:47.909518 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 16942
I20240501 19:57:47.910624 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap starting.
I20240501 19:57:47.925326 17009 log.cc:824] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:47.999351 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=156 overwritten=0 applied=156 ignored=0} inserts{seen=104 ignored=0} mutations{seen=133 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:47.999858 17009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Bootstrap complete.
I20240501 19:57:48.002875 17009 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:48.003203 17009 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.003648 17009 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 6.156, Last appended by leader: 156, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:48.004056 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:48.004174 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:48.004709 17009 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:48.011729 17028 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:48.011816 17028 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: loading cluster ID for follower catalog manager: success
I20240501 19:57:48.015935 17028 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: acquiring CA information for follower catalog manager: success
I20240501 19:57:48.016533 16942 master_runner.cc:186] Error getting master registration for 127.14.96.61:46321: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111),
I20240501 19:57:48.018787 16942 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:48.019364 17028 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:48.171672 15647 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:48.173628 16973 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:52647; Asking this server to re-register.
I20240501 19:57:48.174216 15647 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:48.175729 16973 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:48.191486 15511 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:48.192579 16973 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:41491; Asking this server to re-register.
I20240501 19:57:48.193006 15511 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:48.194000 16973 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:48.298628 15373 heartbeater.cc:347] Connected to a master server at 127.14.96.62:37747
I20240501 19:57:48.299661 16973 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:40803; Asking this server to re-register.
I20240501 19:57:48.300061 15373 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:48.301050 16973 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.460229 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:48.460467 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:48.461830 17014 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:48.462334 16957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
W20240501 19:57:48.463100 16957 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:48.465445 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 156 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:48.465597 16848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f177158f524f0183f6eec721ea83e4 in term 6.
I20240501 19:57:48.465929 16958 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:48.466181 17014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240501 19:57:48.466254 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:48.466302 17014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 19:57:48.468394 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:48.469095 17014 leader_election.cc:288] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 election: Requested vote from peers 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
I20240501 19:57:48.469188 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f177158f524f0183f6eec721ea83e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 156 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763"
I20240501 19:57:48.469352 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 6 FOLLOWER]: Advancing to term 7
W20240501 19:57:48.470225 16957 leader_election.cc:334] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:48.470391 16848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f177158f524f0183f6eec721ea83e4 in term 7.
I20240501 19:57:48.470661 16958 leader_election.cc:302] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4; no voters: 72d8e36923de4d1e8a6de93e73c84658
I20240501 19:57:48.470840 17014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 19:57:48.471026 17014 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [term 7 LEADER]: Becoming Leader. State: Replica: e9f177158f524f0183f6eec721ea83e4, State: Running, Role: LEADER
I20240501 19:57:48.471385 17014 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 156, Committed index: 156, Last appended: 6.156, Last appended by leader: 156, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:48.472986 17036 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f177158f524f0183f6eec721ea83e4. Latest consensus state: current_term: 7 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:48.473088 17036 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:48.473405 17038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:48.475370 17038 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:48.475934 17038 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:48.476107 17038 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:48.476274 17038 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:48.476436 17038 catalog_manager.cc:669] Loaded metadata for table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:48.476649 17038 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:48.476894 17038 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:48.477102 17038 catalog_manager.cc:669] Loaded metadata for table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:48.477308 17038 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:48.477588 17038 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:48.477783 17038 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:48.477980 17038 catalog_manager.cc:669] Loaded metadata for table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:48.478178 17038 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:48.478467 17038 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:48.478664 17038 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:48.478860 17038 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:48.479142 17038 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:48.479336 17038 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:48.479550 17038 catalog_manager.cc:669] Loaded metadata for table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:48.482028 17038 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:48.482504 17038 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:48.482896 17038 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:48.483155 17038 tablet_loader.cc:96] loaded metadata for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:48.483402 17038 tablet_loader.cc:96] loaded metadata for tablet 568a6e167a0d4040aac09233e544c73d (table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1])
I20240501 19:57:48.483558 17038 tablet_loader.cc:96] loaded metadata for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71])
I20240501 19:57:48.483704 17038 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:48.484059 17038 tablet_loader.cc:96] loaded metadata for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:48.484266 17038 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:48.484431 17038 tablet_loader.cc:96] loaded metadata for tablet 956a62d864b14bf7b433c7fd7c335d1e (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:48.484565 17038 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:48.484890 17038 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:48.485119 17038 tablet_loader.cc:96] loaded metadata for tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:48.485252 17038 tablet_loader.cc:96] loaded metadata for tablet be935df388a842d396a805bfc892496d (table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0])
I20240501 19:57:48.485417 17038 tablet_loader.cc:96] loaded metadata for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:48.485661 17038 tablet_loader.cc:96] loaded metadata for tablet db41bcd8b98a45a3b384f1642ae4396e (table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af])
I20240501 19:57:48.486145 17038 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:48.486471 17038 tablet_loader.cc:96] loaded metadata for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:48.486635 17038 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:48.486793 17038 tablet_loader.cc:96] loaded metadata for tablet fe8ac073d20b45138610f9f4ffad1138 (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:48.486949 17038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:48.487928 17038 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:48.487982 17038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:48.489073 17038 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:48.489986 17038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Loaded TSK: 0
I20240501 19:57:48.490710 17038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:48.491633 17038 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:48.492523 17038 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 45
I20240501 19:57:48.513741 16973 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 from {username='slave'} at 127.0.0.1:42358
I20240501 19:57:48.514325 16967 master_service.cc:946] ReplaceTablet: received request to replace tablet be78b5885c0a43ffa72eebb60579b578 from {username='slave'} at 127.0.0.1:42358
I20240501 19:57:48.514200 16972 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_189735560cf84873a69187bf084e7654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.514750 16971 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42358:
table { table_name: "default.Table_3fd15f4a5d9f4fe2a7062c4fb1c53392" } modify_external_catalogs: true
I20240501 19:57:48.514897 16971 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42358:
table { table_name: "default.Table_3fd15f4a5d9f4fe2a7062c4fb1c53392" } modify_external_catalogs: true
I20240501 19:57:48.515008 16970 master_service.cc:946] ReplaceTablet: received request to replace tablet e896ddb02b824d6c9c5519c457f537bc from {username='slave'} at 127.0.0.1:42358
I20240501 19:57:48.515574 16970 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_84bf2da3cdaf45fa84c6099c013783e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.515692 16966 master_service.cc:946] ReplaceTablet: received request to replace tablet 75b4c1663db04508b03bb074ed9fc788 from {username='slave'} at 127.0.0.1:42358
I20240501 19:57:48.515841 16968 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42358:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:48.515950 16968 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42358:
table { table_name: "default.Table_3df0bbd8beda416987f50555069e12d2" } modify_external_catalogs: true
I20240501 19:57:48.516068 16966 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_abc8d72a6cc74062a62861c18c185a62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.516175 16964 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_a62bb6dcb7644aa4bb9008b9f9a196dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.516315 16969 master_service.cc:946] ReplaceTablet: received request to replace tablet fe8ac073d20b45138610f9f4ffad1138 from {username='slave'} at 127.0.0.1:42358
I20240501 19:57:48.516619 16965 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42358:
table { table_name: "default.Table_8d1020ff0b064f7199ad44d673369f79" } new_table_name: "default.Table_5dc76a7d16884cc59719736ac8692923" modify_external_catalogs: true
I20240501 19:57:48.518806 16848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Refusing update from remote peer e9f177158f524f0183f6eec721ea83e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 156. Preceding OpId from leader: term: 7 index: 158. (index mismatch)
I20240501 19:57:48.519973 17036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 157, Last known committed idx: 156, Time since last communication: 0.000s
I20240501 19:57:48.522351 17043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f177158f524f0183f6eec721ea83e4. Latest consensus state: current_term: 7 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:48.522910 17027 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:48.523240 17027 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
I20240501 19:57:48.523263 17043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
F20240501 19:57:48.523330 17027 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
*** Check failure stack trace: ***
W20240501 19:57:48.524430 16957 consensus_peers.cc:485] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 -> Peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Couldn't send request to peer 72d8e36923de4d1e8a6de93e73c84658. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:48.524480 17014 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 7 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:48.524596 17014 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:48.524700 17039 mvcc.cc:205] Tried to move back new op lower bound from 7022974847051227136 to 7022974846861193216. Current Snapshot: MvccSnapshot[applied={T|T < 7022974847045332992}]
I20240501 19:57:48.525504 17043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 7 leader_uuid: "e9f177158f524f0183f6eec721ea83e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:48.525667 17043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:48.525579 17044 mvcc.cc:205] Tried to move back new op lower bound from 7022974847051227136 to 7022974846861193216. Current Snapshot: MvccSnapshot[applied={T|T < 7022974847045332992}]
I20240501 19:57:48.526476 16969 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Sending DeleteTablet for 3 replicas of tablet fe8ac073d20b45138610f9f4ffad1138
I20240501 19:57:48.526476 16973 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Sending DeleteTablet for 3 replicas of tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3
@ 0x7f7582d07dcd google::LogMessage::Fail() at ??:0
I20240501 19:57:48.528424 16967 catalog_manager.cc:5771] T 00000000000000000000000000000000 P e9f177158f524f0183f6eec721ea83e4: Sending DeleteTablet for 3 replicas of tablet be78b5885c0a43ffa72eebb60579b578
I20240501 19:57:48.529783 16969 catalog_manager.cc:6554] ReplaceTablet: tablet fe8ac073d20b45138610f9f4ffad1138 deleted and replaced by tablet d42e1516bbc24af8b461d97ac766d28a
I20240501 19:57:48.529839 16973 catalog_manager.cc:6554] ReplaceTablet: tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 deleted and replaced by tablet 45593e3420774514b6198e32562ee5d7
@ 0x7f7582d0bb93 google::LogMessage::SendToLog() at ??:0
I20240501 19:57:48.530105 16973 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_206463f213f945b2a74cbbcddbada5e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.530148 16969 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_7a325fdcd5224571b886231cf98de803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:48.529785 16967 catalog_manager.cc:6554] ReplaceTablet: tablet be78b5885c0a43ffa72eebb60579b578 deleted and replaced by tablet 8abc2278c6da432caba27a0dbe644573
I20240501 19:57:48.531401 16967 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42358:
name: "default.Table_a283fbed2b5748f1b138b73c95bb2dc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
@ 0x7f7582d077cc google::LogMessage::Flush() at ??:0
@ 0x7f7582d07a11 google::LogMessage::~LogMessage() at ??:0
I20240501 19:57:48.537775 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8ac073d20b45138610f9f4ffad1138 with delete_type TABLET_DATA_DELETED (replaced by tablet d42e1516bbc24af8b461d97ac766d28a) from {username='slave'} at 127.0.0.1:34166
I20240501 19:57:48.537565 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 45593e3420774514b6198e32562ee5d7) from {username='slave'} at 127.0.0.1:34166
I20240501 19:57:48.537710 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet be78b5885c0a43ffa72eebb60579b578 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abc2278c6da432caba27a0dbe644573) from {username='slave'} at 127.0.0.1:34166
I20240501 19:57:48.541430 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet be78b5885c0a43ffa72eebb60579b578 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abc2278c6da432caba27a0dbe644573) from {username='slave'} at 127.0.0.1:58950
I20240501 19:57:48.541519 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8ac073d20b45138610f9f4ffad1138 with delete_type TABLET_DATA_DELETED (replaced by tablet d42e1516bbc24af8b461d97ac766d28a) from {username='slave'} at 127.0.0.1:58950
@ 0x7f7587c334f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240501 19:57:48.541698 15577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 45593e3420774514b6198e32562ee5d7) from {username='slave'} at 127.0.0.1:58950
I20240501 19:57:48.544168 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 8abc2278c6da432caba27a0dbe644573 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.544502 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abc2278c6da432caba27a0dbe644573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.545785 15444 tablet_service.cc:1467] Processing CreateTablet for tablet 45593e3420774514b6198e32562ee5d7 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.546103 15444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45593e3420774514b6198e32562ee5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.546036 15576 tablet_service.cc:1467] Processing CreateTablet for tablet 45593e3420774514b6198e32562ee5d7 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f7587c325ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240501 19:57:48.546365 15576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45593e3420774514b6198e32562ee5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.547262 15442 tablet_service.cc:1467] Processing CreateTablet for tablet d42e1516bbc24af8b461d97ac766d28a (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.547488 15442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e1516bbc24af8b461d97ac766d28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.548985 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 8abc2278c6da432caba27a0dbe644573 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.549008 15577 tablet_service.cc:1467] Processing CreateTablet for tablet d42e1516bbc24af8b461d97ac766d28a (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.549091 17059 tablet_replica.cc:330] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:48.549224 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abc2278c6da432caba27a0dbe644573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.549240 17059 raft_consensus.cc:2229] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.549271 15577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e1516bbc24af8b461d97ac766d28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.549356 17059 raft_consensus.cc:2258] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f7587c31d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240501 19:57:48.549759 17059 ts_tablet_manager.cc:1875] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.550030 17058 tablet_replica.cc:330] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:48.550158 17058 raft_consensus.cc:2229] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.550266 17058 raft_consensus.cc:2258] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.550637 17058 ts_tablet_manager.cc:1875] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7587c36fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:48.553236 15303 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8ac073d20b45138610f9f4ffad1138 with delete_type TABLET_DATA_DELETED (replaced by tablet d42e1516bbc24af8b461d97ac766d28a) from {username='slave'} at 127.0.0.1:42564
@ 0x55753d32275c std::function<>::operator()() at ??:0
I20240501 19:57:48.553619 15305 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 with delete_type TABLET_DATA_DELETED (replaced by tablet 45593e3420774514b6198e32562ee5d7) from {username='slave'} at 127.0.0.1:42564
I20240501 19:57:48.553660 17058 ts_tablet_manager.cc:1888] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.553663 17059 ts_tablet_manager.cc:1888] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.553735 17058 log.cc:1196] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/5d2a3b7c06ec4067bc4fb9e8f91ffad3
I20240501 19:57:48.553735 17059 log.cc:1196] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/be78b5885c0a43ffa72eebb60579b578
I20240501 19:57:48.553752 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet be78b5885c0a43ffa72eebb60579b578 with delete_type TABLET_DATA_DELETED (replaced by tablet 8abc2278c6da432caba27a0dbe644573) from {username='slave'} at 127.0.0.1:42564
I20240501 19:57:48.554013 17058 ts_tablet_manager.cc:1909] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:48.554013 17059 ts_tablet_manager.cc:1909] T be78b5885c0a43ffa72eebb60579b578 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:48.554100 15302 tablet_service.cc:1467] Processing CreateTablet for tablet 8abc2278c6da432caba27a0dbe644573 (DEFAULT_TABLE table=default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.554438 15302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abc2278c6da432caba27a0dbe644573. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.554592 15301 tablet_service.cc:1467] Processing CreateTablet for tablet d42e1516bbc24af8b461d97ac766d28a (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.554828 17059 tablet_replica.cc:330] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:48.554867 15301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e1516bbc24af8b461d97ac766d28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.554963 17059 raft_consensus.cc:2229] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.555073 17059 raft_consensus.cc:2258] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.555474 17059 ts_tablet_manager.cc:1875] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f7584148807 kudu::Thread::SuperviseThread() at ??:0
I20240501 19:57:48.557066 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 45593e3420774514b6198e32562ee5d7 (DEFAULT_TABLE table=default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:48.557348 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45593e3420774514b6198e32562ee5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:48.557554 17059 ts_tablet_manager.cc:1888] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.557622 17059 log.cc:1196] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/5d2a3b7c06ec4067bc4fb9e8f91ffad3
I20240501 19:57:48.557607 16959 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet be78b5885c0a43ffa72eebb60579b578 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]) successfully deleted
I20240501 19:57:48.558048 17059 ts_tablet_manager.cc:1909] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:48.558094 17063 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:48.558698 17059 tablet_replica.cc:330] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:48.558813 17059 raft_consensus.cc:2229] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
@ 0x7f75849866db start_thread at ??:0
I20240501 19:57:48.558845 16959 catalog_manager.cc:4830] TS 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023): tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]) successfully deleted
I20240501 19:57:48.558916 17059 raft_consensus.cc:2258] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.559286 17059 ts_tablet_manager.cc:1875] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.559958 17063 tablet_bootstrap.cc:654] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.560829 17058 tablet_replica.cc:330] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:48.560940 16960 catalog_manager.cc:4830] TS 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181): tablet 5d2a3b7c06ec4067bc4fb9e8f91ffad3 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]) successfully deleted
I20240501 19:57:48.560995 17058 raft_consensus.cc:2229] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:48.561218 17058 raft_consensus.cc:2258] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.561370 17063 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:48.561465 17063 ts_tablet_manager.cc:1367] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
@ 0x7f7581ebe71f clone at ??:0
I20240501 19:57:48.562028 17063 raft_consensus.cc:348] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.562151 17063 raft_consensus.cc:374] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.562191 17063 raft_consensus.cc:729] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.561883 17059 ts_tablet_manager.cc:1888] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:48.562326 17059 log.cc:1196] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/fe8ac073d20b45138610f9f4ffad1138
I20240501 19:57:48.562338 17063 consensus_queue.cc:260] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.562609 17059 ts_tablet_manager.cc:1909] T fe8ac073d20b45138610f9f4ffad1138 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:48.562748 17063 ts_tablet_manager.cc:1398] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:48.562968 17063 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:48.563359 17058 ts_tablet_manager.cc:1875] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.564095 17063 tablet_bootstrap.cc:654] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.565012 17062 tablet_replica.cc:330] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:48.565132 17062 raft_consensus.cc:2229] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.565249 17062 raft_consensus.cc:2258] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.565591 17062 ts_tablet_manager.cc:1875] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.566872 17058 ts_tablet_manager.cc:1888] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:48.566960 17058 log.cc:1196] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/fe8ac073d20b45138610f9f4ffad1138
I20240501 19:57:48.567242 17058 ts_tablet_manager.cc:1909] T fe8ac073d20b45138610f9f4ffad1138 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:48.567600 17060 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:48.568074 17062 ts_tablet_manager.cc:1888] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:48.568111 17063 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:48.568140 17062 log.cc:1196] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/fe8ac073d20b45138610f9f4ffad1138
I20240501 19:57:48.568197 17063 ts_tablet_manager.cc:1367] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:48.568461 17062 ts_tablet_manager.cc:1909] T fe8ac073d20b45138610f9f4ffad1138 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:48.568724 17063 raft_consensus.cc:348] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.568825 17063 raft_consensus.cc:374] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.568885 17063 raft_consensus.cc:729] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.568909 17060 tablet_bootstrap.cc:654] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.569044 17063 consensus_queue.cc:260] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.569247 17062 tablet_replica.cc:330] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:48.569424 17062 raft_consensus.cc:2229] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:48.569607 17062 raft_consensus.cc:2258] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.569965 17061 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:48.570608 17063 ts_tablet_manager.cc:1398] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:48.571084 17061 tablet_bootstrap.cc:654] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.571429 17063 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:48.572285 17058 tablet_replica.cc:330] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:48.572331 17063 tablet_bootstrap.cc:654] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.572425 17058 raft_consensus.cc:2229] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:48.572634 17058 raft_consensus.cc:2258] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.573076 17058 ts_tablet_manager.cc:1875] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.573073 17062 ts_tablet_manager.cc:1875] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.575471 17058 ts_tablet_manager.cc:1888] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.575541 17058 log.cc:1196] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/be78b5885c0a43ffa72eebb60579b578
I20240501 19:57:48.575639 17062 ts_tablet_manager.cc:1888] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.575718 17062 log.cc:1196] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/5d2a3b7c06ec4067bc4fb9e8f91ffad3
I20240501 19:57:48.575799 17058 ts_tablet_manager.cc:1909] T be78b5885c0a43ffa72eebb60579b578 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:48.576009 17062 ts_tablet_manager.cc:1909] T 5d2a3b7c06ec4067bc4fb9e8f91ffad3 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:48.576596 17062 tablet_replica.cc:330] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:48.576726 17062 raft_consensus.cc:2229] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:48.576828 17062 raft_consensus.cc:2258] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:48.576954 17060 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:48.577055 17060 ts_tablet_manager.cc:1367] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240501 19:57:48.577486 17061 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:48.577566 17061 ts_tablet_manager.cc:1367] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:57:48.577636 17063 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:48.577592 17060 raft_consensus.cc:348] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.577703 17063 ts_tablet_manager.cc:1367] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:57:48.577708 17060 raft_consensus.cc:374] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.577739 17060 raft_consensus.cc:729] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.577883 17060 consensus_queue.cc:260] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.578197 17060 ts_tablet_manager.cc:1398] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:48.578145 17063 raft_consensus.cc:348] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.578254 17063 raft_consensus.cc:374] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.578286 17063 raft_consensus.cc:729] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.578392 17060 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:48.578397 17063 consensus_queue.cc:260] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.579103 17061 raft_consensus.cc:348] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.579363 17061 raft_consensus.cc:374] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.579409 17061 raft_consensus.cc:729] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.579566 17061 consensus_queue.cc:260] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.580070 17060 tablet_bootstrap.cc:654] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.580611 17061 ts_tablet_manager.cc:1398] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:48.580798 17061 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:48.582006 17061 tablet_bootstrap.cc:654] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.582224 17060 tablet_bootstrap.cc:492] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:48.582316 17060 ts_tablet_manager.cc:1367] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240501 19:57:48.582798 17060 raft_consensus.cc:348] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.582906 17060 raft_consensus.cc:374] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.582937 17060 raft_consensus.cc:729] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.583065 17060 consensus_queue.cc:260] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.583367 17060 ts_tablet_manager.cc:1398] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:48.583539 17063 ts_tablet_manager.cc:1398] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:57:48.583562 17060 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:48.584482 17062 ts_tablet_manager.cc:1875] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:48.585361 17060 tablet_bootstrap.cc:654] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.586426 17060 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:48.586500 17060 ts_tablet_manager.cc:1367] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:57:48.586962 17060 raft_consensus.cc:348] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.587019 17062 ts_tablet_manager.cc:1888] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:48.587062 17060 raft_consensus.cc:374] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.587091 17062 log.cc:1196] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/be78b5885c0a43ffa72eebb60579b578
I20240501 19:57:48.587095 17060 raft_consensus.cc:729] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.587219 17060 consensus_queue.cc:260] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.587355 17062 ts_tablet_manager.cc:1909] T be78b5885c0a43ffa72eebb60579b578 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:48.587508 17060 ts_tablet_manager.cc:1398] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:48.589859 17061 tablet_bootstrap.cc:492] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:48.589963 17061 ts_tablet_manager.cc:1367] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:48.590536 17061 raft_consensus.cc:348] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.590672 17061 raft_consensus.cc:374] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.590711 17061 raft_consensus.cc:729] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.590920 17061 consensus_queue.cc:260] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.591279 17061 ts_tablet_manager.cc:1398] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:48.592149 17061 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:48.593690 17061 tablet_bootstrap.cc:654] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:48.594952 17061 tablet_bootstrap.cc:492] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:48.595052 17061 ts_tablet_manager.cc:1367] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:48.595623 17061 raft_consensus.cc:348] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.595742 17061 raft_consensus.cc:374] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:48.595777 17061 raft_consensus.cc:729] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:48.595896 17061 consensus_queue.cc:260] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.596238 17061 ts_tablet_manager.cc:1398] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:48.728140 17032 raft_consensus.cc:482] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:48.728359 17032 raft_consensus.cc:504] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.729069 17032 leader_election.cc:288] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.729562 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45593e3420774514b6198e32562ee5d7" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:48.729620 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45593e3420774514b6198e32562ee5d7" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:48.729725 15326 raft_consensus.cc:2454] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:48.729771 15598 raft_consensus.cc:2454] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:48.730113 15399 leader_election.cc:302] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:48.730306 17032 raft_consensus.cc:2781] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:48.730382 17032 raft_consensus.cc:482] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:48.730419 17032 raft_consensus.cc:3037] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.731441 16937 raft_consensus.cc:482] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:48.731542 16937 raft_consensus.cc:504] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.732080 16937 leader_election.cc:288] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.732547 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abc2278c6da432caba27a0dbe644573" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:48.732600 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abc2278c6da432caba27a0dbe644573" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:48.732688 15326 raft_consensus.cc:2454] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:48.732728 15463 raft_consensus.cc:2454] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:48.733014 15533 leader_election.cc:302] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:48.733175 17032 raft_consensus.cc:504] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.733302 16937 raft_consensus.cc:2781] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:48.733361 16937 raft_consensus.cc:482] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:48.733397 16937 raft_consensus.cc:3037] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.735459 16937 raft_consensus.cc:504] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.735858 16937 leader_election.cc:288] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.737007 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abc2278c6da432caba27a0dbe644573" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:48.737134 15326 raft_consensus.cc:3037] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.737141 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abc2278c6da432caba27a0dbe644573" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:48.737250 15463 raft_consensus.cc:3037] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.738152 15463 raft_consensus.cc:2454] T 8abc2278c6da432caba27a0dbe644573 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:48.738152 15326 raft_consensus.cc:2454] T 8abc2278c6da432caba27a0dbe644573 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:48.738459 15533 leader_election.cc:302] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:48.738619 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45593e3420774514b6198e32562ee5d7" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:48.738725 15326 raft_consensus.cc:3037] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.738840 16937 raft_consensus.cc:2781] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:48.739061 16937 raft_consensus.cc:686] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:48.739249 16937 consensus_queue.cc:237] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.739585 15326 raft_consensus.cc:2454] T 45593e3420774514b6198e32562ee5d7 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:48.739846 17032 leader_election.cc:288] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.739926 15397 leader_election.cc:302] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:48.740038 15598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45593e3420774514b6198e32562ee5d7" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:48.740145 15598 raft_consensus.cc:3037] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.740166 17032 raft_consensus.cc:2781] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:48.740340 17032 raft_consensus.cc:686] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:48.740494 17032 consensus_queue.cc:237] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.742357 15598 raft_consensus.cc:2454] T 45593e3420774514b6198e32562ee5d7 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
W20240501 19:57:48.745271 15397 connection.cc:535] server connection from 127.0.0.1:34166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745242 15533 connection.cc:535] server connection from 127.0.0.1:58950 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745246 15260 connection.cc:535] server connection from 127.0.0.1:42564 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745252 15261 connection.cc:535] client connection to 127.14.96.62:37747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745420 15399 connection.cc:535] client connection to 127.14.96.62:37747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745754 15535 connection.cc:535] client connection to 127.14.96.62:37747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.745878 15655 connection.cc:535] client connection to 127.14.96.62:37747 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:48.746305 15655 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.62:37747)
W20240501 19:57:48.746652 15655 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.62:37747)
W20240501 19:57:48.746734 15655 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.62:37747)
W20240501 19:57:48.746814 15655 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.62:37747)
I20240501 19:57:48.852201 17083 raft_consensus.cc:482] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:48.852363 17083 raft_consensus.cc:504] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.852820 17083 leader_election.cc:288] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.853289 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e1516bbc24af8b461d97ac766d28a" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:48.853308 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e1516bbc24af8b461d97ac766d28a" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:48.853431 15463 raft_consensus.cc:2454] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:48.853469 15326 raft_consensus.cc:2454] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:48.853767 15536 leader_election.cc:302] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:48.853953 17083 raft_consensus.cc:2781] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:48.854017 17083 raft_consensus.cc:482] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:48.854070 17083 raft_consensus.cc:3037] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.854991 17083 raft_consensus.cc:504] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:48.855340 17083 leader_election.cc:288] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:48.855698 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e1516bbc24af8b461d97ac766d28a" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:48.855703 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e1516bbc24af8b461d97ac766d28a" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:48.855830 15463 raft_consensus.cc:3037] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.855831 15326 raft_consensus.cc:3037] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:48.856675 15463 raft_consensus.cc:2454] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:48.856683 15326 raft_consensus.cc:2454] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:48.856994 15533 leader_election.cc:302] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:48.857244 17083 raft_consensus.cc:2781] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:48.857367 17083 raft_consensus.cc:686] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:48.857537 17083 consensus_queue.cc:237] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:49.043689 17089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:49.043818 17089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:49.044286 17089 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:49.045445 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:49.045660 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:49.045732 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:49.045884 17089 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240501 19:57:49.140035 17083 consensus_queue.cc:1035] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.144596 17083 consensus_queue.cc:1035] T 8abc2278c6da432caba27a0dbe644573 P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.236112 17083 consensus_queue.cc:1035] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.239537 16937 consensus_queue.cc:1035] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.256388 17032 consensus_queue.cc:1035] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.260334 16806 consensus_queue.cc:1035] T 45593e3420774514b6198e32562ee5d7 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:49.647847 17105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:49.647980 17105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:49.648394 17105 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:49.649652 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:49.649755 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:49.649837 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:49.650002 17105 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240501 19:57:49.792236 16807 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:57:49.792361 16807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:50.688469 17120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:50.688601 17120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:50.689121 17120 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:50.690320 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:50.690435 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:50.690516 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:50.690649 17120 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240501 19:57:51.129096 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--webserver_interface=127.14.96.61
--webserver_port=42603
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240501 19:57:51.228700 17132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:51.228864 17132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:51.229295 17132 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:51.229838 16822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 19:57:51.230544 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:51.230782 16822 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321): Network error: Client connection negotiation failed: client connection to 127.14.96.61:46321: connect: Connection refused (error 111)
I20240501 19:57:51.230870 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763; no voters: 72d8e36923de4d1e8a6de93e73c84658, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:51.231005 17132 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240501 19:57:51.245550 17131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:51.245791 17131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:51.245847 17131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:51.249171 17131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:51.249238 17131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:51.249259 17131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:51.249279 17131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:51.253315 17131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.96.61
--webserver_port=42603
--never_fsync=true
--heap_profile_path=/tmp/kudu.17131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:51.254343 17131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:51.255506 17131 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:51.263840 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:51.264434 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:51.265041 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:51.265703 17131 server_base.cc:1034] running on GCE node
I20240501 19:57:51.266233 17131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:51.267177 17131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:51.268383 17131 hybrid_clock.cc:648] HybridClock initialized: now 1714593471268367 us; error 29 us; skew 500 ppm
I20240501 19:57:51.270715 17131 webserver.cc:460] Webserver started at http://127.14.96.61:42603/ using document root <none> and password file <none>
I20240501 19:57:51.271310 17131 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:51.271373 17131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:51.275185 17131 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240501 19:57:51.277500 17151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:51.278597 17131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:51.278740 17131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:51.279186 17131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:51.306953 17131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:51.307647 17131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:51.307832 17131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:51.314761 17131 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.61:46321
I20240501 19:57:51.314785 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.61:46321 every 8 connection(s)
I20240501 19:57:51.315860 17131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
I20240501 19:57:51.316078 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 17131
I20240501 19:57:51.321012 17207 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:51.323068 17207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap starting.
I20240501 19:57:51.336299 17207 log.cc:824] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:51.343851 15512 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:51.410693 17207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=151 ignored=0} inserts{seen=103 ignored=0} mutations{seen=131 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:51.411166 17207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap complete.
I20240501 19:57:51.414736 17207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:51.415078 17207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Initialized, Role: FOLLOWER
I20240501 19:57:51.415545 17207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 3.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:51.416014 17214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:51.416136 17214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:51.416647 17207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:51.423756 17228 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:51.423822 17228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: loading cluster ID for follower catalog manager: success
I20240501 19:57:51.422049 17131 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:57:51.424459 17131 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:51.427742 17228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: acquiring CA information for follower catalog manager: success
I20240501 19:57:51.430648 17228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:51.768433 15374 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:51.770493 17171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:38511; Asking this server to re-register.
I20240501 19:57:51.771018 15374 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:51.772347 17171 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:51.797782 15648 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:51.799073 17171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:43837; Asking this server to re-register.
I20240501 19:57:51.799598 15648 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:51.800561 17171 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:51.850250 17171 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:48155; Asking this server to re-register.
I20240501 19:57:51.850805 15512 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:51.851882 17171 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:51.947695 17236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:51.947919 17236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:51.949255 17236 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:51.949955 17154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:51.950798 17154 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:51.953366 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72d8e36923de4d1e8a6de93e73c84658" candidate_term: 4 candidate_status { last_received { term: 3 index: 151 } } ignore_live_leader: false dest_uuid: "62106a6dbb3846df9d3b29ab99fb3763" is_pre_election: true
I20240501 19:57:51.953563 16848 raft_consensus.cc:2354] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72d8e36923de4d1e8a6de93e73c84658 for earlier term 4. Current term is 7.
I20240501 19:57:51.954128 17153 leader_election.cc:398] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72d8e36923de4d1e8a6de93e73c84658 for earlier term 4. Current term is 7.
I20240501 19:57:51.954205 17153 leader_election.cc:401] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240501 19:57:51.954427 17236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 3 FOLLOWER]: Advancing to term 7
I20240501 19:57:51.956403 17236 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 7 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72d8e36923de4d1e8a6de93e73c84658 for earlier term 4. Current term is 7.
I20240501 19:57:52.602296 17242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:52.602439 17242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:52.602911 17242 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
W20240501 19:57:52.604377 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:52.607070 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 8 candidate_status { last_received { term: 7 index: 161 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658" is_pre_election: true
I20240501 19:57:52.607358 17181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 7.
I20240501 19:57:52.607748 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658; no voters: e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:52.607967 17242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240501 19:57:52.608033 17242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e9f177158f524f0183f6eec721ea83e4)
I20240501 19:57:52.608063 17242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 7 FOLLOWER]: Advancing to term 8
I20240501 19:57:52.608987 17242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:52.609362 17242 leader_election.cc:288] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 election: Requested vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 72d8e36923de4d1e8a6de93e73c84658 (127.14.96.61:46321)
I20240501 19:57:52.609802 17181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "62106a6dbb3846df9d3b29ab99fb3763" candidate_term: 8 candidate_status { last_received { term: 7 index: 161 } } ignore_live_leader: false dest_uuid: "72d8e36923de4d1e8a6de93e73c84658"
I20240501 19:57:52.609943 17181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 7 FOLLOWER]: Advancing to term 8
W20240501 19:57:52.610574 16824 leader_election.cc:334] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:52.611215 17181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62106a6dbb3846df9d3b29ab99fb3763 in term 8.
I20240501 19:57:52.611528 16822 leader_election.cc:302] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 62106a6dbb3846df9d3b29ab99fb3763, 72d8e36923de4d1e8a6de93e73c84658; no voters: e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:52.611696 17242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 19:57:52.611955 17242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [term 8 LEADER]: Becoming Leader. State: Replica: 62106a6dbb3846df9d3b29ab99fb3763, State: Running, Role: LEADER
I20240501 19:57:52.612272 17242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 161, Committed index: 161, Last appended: 7.161, Last appended by leader: 161, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:52.614001 17247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.614112 17247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:52.614395 17249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:52.616392 17249 catalog_manager.cc:669] Loaded metadata for table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]
I20240501 19:57:52.617105 17249 catalog_manager.cc:669] Loaded metadata for table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525]
I20240501 19:57:52.617311 17249 catalog_manager.cc:669] Loaded metadata for table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2]
I20240501 19:57:52.617511 17249 catalog_manager.cc:669] Loaded metadata for table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b]
I20240501 19:57:52.617717 17249 catalog_manager.cc:669] Loaded metadata for table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af]
I20240501 19:57:52.617894 17249 catalog_manager.cc:669] Loaded metadata for table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992]
I20240501 19:57:52.618116 17249 catalog_manager.cc:669] Loaded metadata for table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef]
I20240501 19:57:52.618350 17249 catalog_manager.cc:669] Loaded metadata for table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71]
I20240501 19:57:52.618561 17249 catalog_manager.cc:669] Loaded metadata for table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]
I20240501 19:57:52.618841 17249 catalog_manager.cc:669] Loaded metadata for table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e]
I20240501 19:57:52.619035 17249 catalog_manager.cc:669] Loaded metadata for table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]
I20240501 19:57:52.619237 17249 catalog_manager.cc:669] Loaded metadata for table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0]
I20240501 19:57:52.619437 17249 catalog_manager.cc:669] Loaded metadata for table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd]
I20240501 19:57:52.619721 17249 catalog_manager.cc:669] Loaded metadata for table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42]
I20240501 19:57:52.619917 17249 catalog_manager.cc:669] Loaded metadata for table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]
I20240501 19:57:52.620110 17249 catalog_manager.cc:669] Loaded metadata for table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d]
I20240501 19:57:52.620390 17249 catalog_manager.cc:669] Loaded metadata for table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5]
I20240501 19:57:52.620586 17249 catalog_manager.cc:669] Loaded metadata for table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0]
I20240501 19:57:52.620779 17249 catalog_manager.cc:669] Loaded metadata for table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1]
I20240501 19:57:52.623064 17249 tablet_loader.cc:96] loaded metadata for tablet 2ed5f03a8ff34913a59c475bd7ec83cf (table default.table_b7b25ac22dba40c9aa0a509437741d50 [id=da34a8c02ca847f394f731cfe148aad0])
I20240501 19:57:52.623363 17249 tablet_loader.cc:96] loaded metadata for tablet 45593e3420774514b6198e32562ee5d7 (table default.table_a4c90268e9694efab8ddbeccf1c2f618 [id=5c9739b72de443feaffb7a56614c8d71])
I20240501 19:57:52.623613 17249 tablet_loader.cc:96] loaded metadata for tablet 4d4a80a5d2f848b78741b35319aa54f5 (table default.table_b16d0f8f6d3341c8bbd83fc19a8c3771 [id=9fbc6c7166d34be6a3d4073bae1995bd])
I20240501 19:57:52.623941 17249 tablet_loader.cc:96] loaded metadata for tablet 4e268796c5da43fa9d65a3b9d0a444d5 (table default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240])
I20240501 19:57:52.624192 17249 tablet_loader.cc:96] loaded metadata for tablet 553a91cc4b2d44feb3d30d3bde9a2919 (table default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527])
I20240501 19:57:52.624441 17249 tablet_loader.cc:96] loaded metadata for tablet 568a6e167a0d4040aac09233e544c73d (table default.table_3fd15f4a5d9f4fe2a7062c4fb1c53392 [id=e72f1d9638c54cdc97e8552ce0152be1])
I20240501 19:57:52.624682 17249 tablet_loader.cc:96] loaded metadata for tablet 71c0ca8638f14b89a2041c10a2e2206d (table default.table_e8a1fac2da6b4b4e9acacc41d87bc317 [id=2576da67068d4a7781445183df23a525])
I20240501 19:57:52.625128 17249 tablet_loader.cc:96] loaded metadata for tablet 89ba8dce2b2f4d4a8a9db68557fdc2d0 (table default.table_43971f18a04d42be9b5f69e32a2031ed [id=4b0264d091414eac90e34e72e5ced992])
I20240501 19:57:52.625304 17249 tablet_loader.cc:96] loaded metadata for tablet 8abc2278c6da432caba27a0dbe644573 (table default.table_3df0bbd8beda416987f50555069e12d2 [id=ce7e23a809264cd280e7fd1acea4eaf5])
I20240501 19:57:52.625550 17249 tablet_loader.cc:96] loaded metadata for tablet 9384878f4f834c9d90db68234d8755ae (table default.table_0c03b0e181d140c88754b558e4fb4393 [id=2434824a54f245fba1c41a6edcc62dad])
I20240501 19:57:52.625710 17249 tablet_loader.cc:96] loaded metadata for tablet 956a62d864b14bf7b433c7fd7c335d1e (table default.table_fb234d1b9ddb449a96ad2affdb553574 [id=c5b2c7b2d6814ad4bbc8c6a6738fbe4d])
I20240501 19:57:52.625870 17249 tablet_loader.cc:96] loaded metadata for tablet 9a6cb71ea54e4cc2985b9d607109aba5 (table default.table_9254e341432a42c29665f212b2913ff6 [id=b17b1d0c163b4ddbad8e7ccd2f0a3d42])
I20240501 19:57:52.626194 17249 tablet_loader.cc:96] loaded metadata for tablet aa19839336314324b4160d1f3c80a92d (table default.table_77b2a55a20c84bc0bea5a2f3df0d5705 [id=31afc8f9a90d41fbaa6f7ee30418792b])
I20240501 19:57:52.626528 17249 tablet_loader.cc:96] loaded metadata for tablet be935df388a842d396a805bfc892496d (table default.table_69448af6833b458b99a6e162b53e46de [id=6bccfb6fee3f49d7bf781dc880460df0])
I20240501 19:57:52.626690 17249 tablet_loader.cc:96] loaded metadata for tablet ca55116fb0b54eb39a00a3fc41211c8f (table default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8])
I20240501 19:57:52.626941 17249 tablet_loader.cc:96] loaded metadata for tablet d42e1516bbc24af8b461d97ac766d28a (table default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28])
I20240501 19:57:52.627099 17249 tablet_loader.cc:96] loaded metadata for tablet db41bcd8b98a45a3b384f1642ae4396e (table default.table_7dee854303424e5db523b5680176f6c5 [id=41fee5d312d44692aab96c44278394af])
I20240501 19:57:52.627573 17249 tablet_loader.cc:96] loaded metadata for tablet ec1ef46612cd4545a753bd4700897535 (table default.table_5481ef3ce5fa4ea2b569d6ac2a569b75 [id=2ae1fe4915db4afd81a5c7a2df5bd0e2])
I20240501 19:57:52.627924 17249 tablet_loader.cc:96] loaded metadata for tablet f8981ca996bc49e2b3a9be9bcdb93bb4 (table default.table_8d1020ff0b064f7199ad44d673369f79 [id=6176d0a75be341ee85f09c5e0ae4074e])
I20240501 19:57:52.628103 17249 tablet_loader.cc:96] loaded metadata for tablet f8c7c7763c08463ba9d1056d39988b4d (table default.table_090e62cd0c6c43eb9c44ada659e01c98 [id=59ee2d94f2c74cccabdf2d145ab79fef])
I20240501 19:57:52.628342 17249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:52.629467 17249 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:52.629520 17249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:52.630560 17249 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:52.631403 17249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Loaded TSK: 0
I20240501 19:57:52.632107 17249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:52.633067 17249 catalog_manager.cc:1527] Loading latest processed Hive Metastore notification log event ID...
I20240501 19:57:52.634243 17249 catalog_manager.cc:5706] Last processed Hive Metastore notification event ID: 45
I20240501 19:57:52.645279 16837 master_service.cc:946] ReplaceTablet: received request to replace tablet ca55116fb0b54eb39a00a3fc41211c8f from {username='slave'} at 127.0.0.1:42596
I20240501 19:57:52.645419 16838 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e268796c5da43fa9d65a3b9d0a444d5 from {username='slave'} at 127.0.0.1:42596
I20240501 19:57:52.646051 16830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:52.646263 16830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_e8a1fac2da6b4b4e9acacc41d87bc317" } modify_external_catalogs: true
I20240501 19:57:52.646209 16832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42596:
name: "default.Table_a82d93e7c3e949e99304ee3a61f2f5e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:52.646539 16831 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:52.646646 16831 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_77b2a55a20c84bc0bea5a2f3df0d5705" } modify_external_catalogs: true
I20240501 19:57:52.646793 16829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5" } new_table_name: "default.Table_557eeb62e4e64cc6aeca0b1882dd3909" modify_external_catalogs: true
I20240501 19:57:52.646834 16833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_090e62cd0c6c43eb9c44ada659e01c98" } new_table_name: "default.Table_f0938e2fdba84a2ca4f93e39ab3e7eac" modify_external_catalogs: true
I20240501 19:57:52.647017 16834 master_service.cc:946] ReplaceTablet: received request to replace tablet d42e1516bbc24af8b461d97ac766d28a from {username='slave'} at 127.0.0.1:42596
I20240501 19:57:52.647742 16835 master_service.cc:946] ReplaceTablet: received request to replace tablet 553a91cc4b2d44feb3d30d3bde9a2919 from {username='slave'} at 127.0.0.1:42596
I20240501 19:57:52.647775 16836 master_service.cc:946] ReplaceTablet: received request to replace tablet ca55116fb0b54eb39a00a3fc41211c8f from {username='slave'} at 127.0.0.1:42596
I20240501 19:57:52.649279 17181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Refusing update from remote peer 62106a6dbb3846df9d3b29ab99fb3763: Log matching property violated. Preceding OpId in replica: term: 3 index: 151. Preceding OpId from leader: term: 8 index: 163. (index mismatch)
I20240501 19:57:52.649896 17247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 162, Last known committed idx: 151, Time since last communication: 0.000s
W20240501 19:57:52.652608 16824 consensus_peers.cc:485] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 -> Peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Couldn't send request to peer e9f177158f524f0183f6eec721ea83e4. Status: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:52.652760 16896 catalog_manager.cc:2706] Deleting table default.table_a283fbed2b5748f1b138b73c95bb2dc3 [id=ca0cb5d4470641eb975fa6f8033d346f] in response to Hive Metastore notification log event 46
W20240501 19:57:52.653086 16896 hms_notification_log_listener.cc:293] Failed to handle Hive Metastore notification: 46 DROP_TABLE default.table_a283fbed2b5748f1b138b73c95bb2dc3: Not found: the table does not exist: table_id: "ca0cb5d4470641eb975fa6f8033d346f" table_name: "default.table_a283fbed2b5748f1b138b73c95bb2dc3"
F20240501 19:57:52.653157 16896 hms_notification_log_listener.cc:264] Received out-of-order notification log event (last processed event ID: 46): 46 ALTER_TABLE default.table_557eeb62e4e64cc6aeca0b1882dd3909
*** Check failure stack trace: ***
@ 0x7f17637d9dcd google::LogMessage::Fail() at ??:0
I20240501 19:57:52.656337 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62106a6dbb3846df9d3b29ab99fb3763. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.656469 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:52.656561 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.656637 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:52.656698 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.656749 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:52.656987 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.657083 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:52.657227 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.657294 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
@ 0x7f17637ddb93 google::LogMessage::SendToLog() at ??:0
I20240501 19:57:52.658727 17242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.658845 17242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:52.659430 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "62106a6dbb3846df9d3b29ab99fb3763" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:52.659525 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:52.661759 17253 mvcc.cc:205] Tried to move back new op lower bound from 7022974863975915520 to 7022974790802329600. Current Snapshot: MvccSnapshot[applied={T|T < 7022974863964368896}]
I20240501 19:57:52.664711 16837 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet ca55116fb0b54eb39a00a3fc41211c8f
@ 0x7f17637d97cc google::LogMessage::Flush() at ??:0
I20240501 19:57:52.665959 16837 catalog_manager.cc:6554] ReplaceTablet: tablet ca55116fb0b54eb39a00a3fc41211c8f deleted and replaced by tablet 7ccb5505f0db4729abf237bc5e6fe492
I20240501 19:57:52.666267 16837 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_b7b25ac22dba40c9aa0a509437741d50" } new_table_name: "default.Table_3846033cb97a4827a54a0f1bcf875141" modify_external_catalogs: true
I20240501 19:57:52.666675 16836 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:52.666769 16836 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_fb234d1b9ddb449a96ad2affdb553574" } modify_external_catalogs: true
I20240501 19:57:52.669062 16834 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet d42e1516bbc24af8b461d97ac766d28a
@ 0x7f17637d9a11 google::LogMessage::~LogMessage() at ??:0
I20240501 19:57:52.669445 16834 catalog_manager.cc:6554] ReplaceTablet: tablet d42e1516bbc24af8b461d97ac766d28a deleted and replaced by tablet 25eef7b37f4d46818c14c0caf8c671ee
I20240501 19:57:52.669562 16838 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 4e268796c5da43fa9d65a3b9d0a444d5
I20240501 19:57:52.669658 16834 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:42596:
table { table_name: "default.Table_b16d0f8f6d3341c8bbd83fc19a8c3771" } new_table_name: "default.Table_ffecde6d8bcf4ef58ab7d34e7904449b" modify_external_catalogs: true
I20240501 19:57:52.669788 16838 catalog_manager.cc:6554] ReplaceTablet: tablet 4e268796c5da43fa9d65a3b9d0a444d5 deleted and replaced by tablet 2b5bc7534f934590805794409e81438e
I20240501 19:57:52.669977 16838 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42596:
name: "default.Table_f18a22bd8291471ba13555510194e198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:52.670656 16835 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 62106a6dbb3846df9d3b29ab99fb3763: Sending DeleteTablet for 3 replicas of tablet 553a91cc4b2d44feb3d30d3bde9a2919
I20240501 19:57:52.670886 16835 catalog_manager.cc:6554] ReplaceTablet: tablet 553a91cc4b2d44feb3d30d3bde9a2919 deleted and replaced by tablet d3a8a2213e8f4f16bf77a9dc19c3cbab
I20240501 19:57:52.671063 16835 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42596:
name: "default.Table_189735560cf84873a69187bf084e7654"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:52.677541 15304 tablet_service.cc:1514] Processing DeleteTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a8a2213e8f4f16bf77a9dc19c3cbab) from {username='slave'} at 127.0.0.1:42578
I20240501 19:57:52.677816 15301 tablet_service.cc:1514] Processing DeleteTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f with delete_type TABLET_DATA_DELETED (replaced by tablet 7ccb5505f0db4729abf237bc5e6fe492) from {username='slave'} at 127.0.0.1:42578
I20240501 19:57:52.677918 15306 tablet_service.cc:1514] Processing DeleteTablet for tablet d42e1516bbc24af8b461d97ac766d28a with delete_type TABLET_DATA_DELETED (replaced by tablet 25eef7b37f4d46818c14c0caf8c671ee) from {username='slave'} at 127.0.0.1:42578
I20240501 19:57:52.677978 15302 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b5bc7534f934590805794409e81438e) from {username='slave'} at 127.0.0.1:42578
@ 0x7f17687054f4 kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20240501 19:57:52.682137 15302 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccb5505f0db4729abf237bc5e6fe492 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.682476 15302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccb5505f0db4729abf237bc5e6fe492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.685431 15577 tablet_service.cc:1514] Processing DeleteTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f with delete_type TABLET_DATA_DELETED (replaced by tablet 7ccb5505f0db4729abf237bc5e6fe492) from {username='slave'} at 127.0.0.1:58962
I20240501 19:57:52.685695 15576 tablet_service.cc:1514] Processing DeleteTablet for tablet d42e1516bbc24af8b461d97ac766d28a with delete_type TABLET_DATA_DELETED (replaced by tablet 25eef7b37f4d46818c14c0caf8c671ee) from {username='slave'} at 127.0.0.1:58962
I20240501 19:57:52.685809 15580 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b5bc7534f934590805794409e81438e) from {username='slave'} at 127.0.0.1:58962
I20240501 19:57:52.685904 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a8a2213e8f4f16bf77a9dc19c3cbab) from {username='slave'} at 127.0.0.1:58962
I20240501 19:57:52.686177 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccb5505f0db4729abf237bc5e6fe492 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.686452 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccb5505f0db4729abf237bc5e6fe492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.687947 15444 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e268796c5da43fa9d65a3b9d0a444d5 with delete_type TABLET_DATA_DELETED (replaced by tablet 2b5bc7534f934590805794409e81438e) from {username='slave'} at 127.0.0.1:34180
I20240501 19:57:52.688189 15443 tablet_service.cc:1514] Processing DeleteTablet for tablet 553a91cc4b2d44feb3d30d3bde9a2919 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a8a2213e8f4f16bf77a9dc19c3cbab) from {username='slave'} at 127.0.0.1:34180
I20240501 19:57:52.688477 15440 tablet_service.cc:1467] Processing CreateTablet for tablet 7ccb5505f0db4729abf237bc5e6fe492 (DEFAULT_TABLE table=default.table_0ef1c033257e4e358ccbe701cce981e2 [id=06697bc6b1904e478b7551bee08787f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.688748 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ccb5505f0db4729abf237bc5e6fe492. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.688953 15442 tablet_service.cc:1514] Processing DeleteTablet for tablet ca55116fb0b54eb39a00a3fc41211c8f with delete_type TABLET_DATA_DELETED (replaced by tablet 7ccb5505f0db4729abf237bc5e6fe492) from {username='slave'} at 127.0.0.1:34180
I20240501 19:57:52.689095 15441 tablet_service.cc:1514] Processing DeleteTablet for tablet d42e1516bbc24af8b461d97ac766d28a with delete_type TABLET_DATA_DELETED (replaced by tablet 25eef7b37f4d46818c14c0caf8c671ee) from {username='slave'} at 127.0.0.1:34180
I20240501 19:57:52.689254 17274 tablet_replica.cc:330] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:52.689464 17274 raft_consensus.cc:2229] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Raft consensus shutting down.
@ 0x7f17687045ee kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20240501 19:57:52.689724 17274 raft_consensus.cc:2258] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.690237 17274 ts_tablet_manager.cc:1875] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f1768703d51 _ZZN4kudu6master30HmsNotificationLogListenerTask4InitEvENKUlvE_clEv at ??:0
I20240501 19:57:52.693128 15302 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.693128 15440 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.693384 15302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.693387 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.694375 17274 ts_tablet_manager.cc:1888] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:52.694456 17274 log.cc:1196] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/553a91cc4b2d44feb3d30d3bde9a2919
I20240501 19:57:52.694761 17274 ts_tablet_manager.cc:1909] T 553a91cc4b2d44feb3d30d3bde9a2919 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
@ 0x7f1768708fc6 _ZNSt17_Function_handlerIFvvEZN4kudu6master30HmsNotificationLogListenerTask4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:52.695969 15306 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5bc7534f934590805794409e81438e (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.696244 15306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5bc7534f934590805794409e81438e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.696439 15440 tablet_service.cc:1467] Processing CreateTablet for tablet 25eef7b37f4d46818c14c0caf8c671ee (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x564c7631d75c std::function<>::operator()() at ??:0
I20240501 19:57:52.696661 15440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eef7b37f4d46818c14c0caf8c671ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.697083 17277 tablet_replica.cc:330] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:52.697381 17277 raft_consensus.cc:2229] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.697530 17277 raft_consensus.cc:2258] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.698014 17277 ts_tablet_manager.cc:1875] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.698542 15301 tablet_service.cc:1467] Processing CreateTablet for tablet 25eef7b37f4d46818c14c0caf8c671ee (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.698772 15301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eef7b37f4d46818c14c0caf8c671ee. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f1764c1a807 kudu::Thread::SuperviseThread() at ??:0
I20240501 19:57:52.700506 15441 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5bc7534f934590805794409e81438e (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.700728 15441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5bc7534f934590805794409e81438e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.700896 17277 ts_tablet_manager.cc:1888] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.700975 17277 log.cc:1196] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/ca55116fb0b54eb39a00a3fc41211c8f
I20240501 19:57:52.701272 17277 ts_tablet_manager.cc:1909] T ca55116fb0b54eb39a00a3fc41211c8f P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:52.701576 17276 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
@ 0x7f17654586db start_thread at ??:0
I20240501 19:57:52.702473 17278 tablet_replica.cc:330] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:52.702651 17278 raft_consensus.cc:2229] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:52.702731 17276 tablet_bootstrap.cc:654] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.702890 17278 raft_consensus.cc:2258] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.703394 17278 ts_tablet_manager.cc:1875] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.703456 15579 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5bc7534f934590805794409e81438e (DEFAULT_TABLE table=default.table_4907ae96e84044d5ace4ea027902eecb [id=661dbbc03c3442a6a48f154754c9b240]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f176299071f clone at ??:0
I20240501 19:57:52.703657 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b5bc7534f934590805794409e81438e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.704242 15580 tablet_service.cc:1467] Processing CreateTablet for tablet 25eef7b37f4d46818c14c0caf8c671ee (DEFAULT_TABLE table=default.table_8fd604fe4a7c4c9c96dd1d1d4ed6a0a5 [id=5ea004652cc64715bc2786eea1248f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.704519 15580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eef7b37f4d46818c14c0caf8c671ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.705948 17278 ts_tablet_manager.cc:1888] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.706024 17278 log.cc:1196] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/4e268796c5da43fa9d65a3b9d0a444d5
I20240501 19:57:52.706326 17278 ts_tablet_manager.cc:1909] T 4e268796c5da43fa9d65a3b9d0a444d5 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:52.707033 17278 tablet_replica.cc:330] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:52.706996 15578 tablet_service.cc:1467] Processing CreateTablet for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab (DEFAULT_TABLE table=default.table_b0d6857034b54b258f3f6d577cd53e13 [id=b299c118e5244019b8ffd6620535d527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:52.707209 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a8a2213e8f4f16bf77a9dc19c3cbab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:52.707191 17278 raft_consensus.cc:2229] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.707319 17278 raft_consensus.cc:2258] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.707752 17278 ts_tablet_manager.cc:1875] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.709504 17274 tablet_replica.cc:330] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:52.709654 17274 raft_consensus.cc:2229] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.709764 17274 raft_consensus.cc:2258] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.709945 17278 ts_tablet_manager.cc:1888] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:52.710006 17278 log.cc:1196] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/553a91cc4b2d44feb3d30d3bde9a2919
I20240501 19:57:52.710184 17274 ts_tablet_manager.cc:1875] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.710261 17278 ts_tablet_manager.cc:1909] T 553a91cc4b2d44feb3d30d3bde9a2919 P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:52.710872 17277 tablet_replica.cc:330] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:52.710999 17277 raft_consensus.cc:2229] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:52.711081 17278 tablet_replica.cc:330] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:52.711198 17278 raft_consensus.cc:2229] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:52.711362 17278 raft_consensus.cc:2258] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.711207 17277 raft_consensus.cc:2258] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.711688 17278 ts_tablet_manager.cc:1875] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.711774 17277 ts_tablet_manager.cc:1875] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.712283 17274 ts_tablet_manager.cc:1888] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.712347 17274 log.cc:1196] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/ca55116fb0b54eb39a00a3fc41211c8f
I20240501 19:57:52.712611 17274 ts_tablet_manager.cc:1909] T ca55116fb0b54eb39a00a3fc41211c8f P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:52.711421 17279 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:52.712953 17280 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:52.713723 17278 ts_tablet_manager.cc:1888] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.713748 17279 tablet_bootstrap.cc:654] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.713785 17278 log.cc:1196] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/ca55116fb0b54eb39a00a3fc41211c8f
I20240501 19:57:52.713868 17280 tablet_bootstrap.cc:654] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.714026 17278 ts_tablet_manager.cc:1909] T ca55116fb0b54eb39a00a3fc41211c8f P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:52.714839 17274 tablet_replica.cc:330] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:52.714970 17274 raft_consensus.cc:2229] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.715080 17274 raft_consensus.cc:2258] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.715121 17276 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:52.715200 17276 ts_tablet_manager.cc:1367] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:57:52.715433 17274 ts_tablet_manager.cc:1875] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.715616 17276 raft_consensus.cc:348] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.715714 17276 raft_consensus.cc:374] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.715744 17276 raft_consensus.cc:729] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.715895 17276 consensus_queue.cc:260] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.716087 17277 ts_tablet_manager.cc:1888] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.716163 17277 log.cc:1196] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/d42e1516bbc24af8b461d97ac766d28a
I20240501 19:57:52.716415 17277 ts_tablet_manager.cc:1909] T d42e1516bbc24af8b461d97ac766d28a P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:52.717190 17277 tablet_replica.cc:330] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:52.717396 17277 raft_consensus.cc:2229] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.717494 17277 raft_consensus.cc:2258] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.717761 17274 ts_tablet_manager.cc:1888] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.717829 17274 log.cc:1196] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/d42e1516bbc24af8b461d97ac766d28a
I20240501 19:57:52.717904 17277 ts_tablet_manager.cc:1875] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.718096 17274 ts_tablet_manager.cc:1909] T d42e1516bbc24af8b461d97ac766d28a P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:52.718837 17280 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:52.718940 17280 ts_tablet_manager.cc:1367] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:57:52.719456 17280 raft_consensus.cc:348] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.719571 17280 raft_consensus.cc:374] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.719610 17280 raft_consensus.cc:729] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.719653 17276 ts_tablet_manager.cc:1398] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:52.719748 17280 consensus_queue.cc:260] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.719985 17277 ts_tablet_manager.cc:1888] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.720045 17277 log.cc:1196] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/4e268796c5da43fa9d65a3b9d0a444d5
I20240501 19:57:52.720088 17280 ts_tablet_manager.cc:1398] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:52.720309 17280 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:52.721297 17280 tablet_bootstrap.cc:654] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.721578 17277 ts_tablet_manager.cc:1909] T 4e268796c5da43fa9d65a3b9d0a444d5 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:52.722206 17277 tablet_replica.cc:330] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: stopping tablet replica
I20240501 19:57:52.722328 17277 raft_consensus.cc:2229] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.722429 17277 raft_consensus.cc:2258] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.722816 17277 ts_tablet_manager.cc:1875] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.723912 17276 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:52.724417 17274 tablet_replica.cc:330] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: stopping tablet replica
I20240501 19:57:52.724542 17274 raft_consensus.cc:2229] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.724638 17274 raft_consensus.cc:2258] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.725050 17274 ts_tablet_manager.cc:1875] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.725245 17277 ts_tablet_manager.cc:1888] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240501 19:57:52.725308 17277 log.cc:1196] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-2/wal/wals/553a91cc4b2d44feb3d30d3bde9a2919
I20240501 19:57:52.725545 17277 ts_tablet_manager.cc:1909] T 553a91cc4b2d44feb3d30d3bde9a2919 P 7f011d422e164c949a9d54d846b7c432: Deleting consensus metadata
I20240501 19:57:52.725544 17276 tablet_bootstrap.cc:654] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.726368 17278 tablet_replica.cc:330] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: stopping tablet replica
I20240501 19:57:52.726531 17278 raft_consensus.cc:2229] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:52.726649 17278 raft_consensus.cc:2258] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:52.727046 17278 ts_tablet_manager.cc:1875] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 19:57:52.727450 17274 ts_tablet_manager.cc:1888] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.727524 17274 log.cc:1196] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-0/wal/wals/4e268796c5da43fa9d65a3b9d0a444d5
I20240501 19:57:52.727797 17274 ts_tablet_manager.cc:1909] T 4e268796c5da43fa9d65a3b9d0a444d5 P 0c8dfc11dc7542f583644b9ac747545d: Deleting consensus metadata
I20240501 19:57:52.729378 17276 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:52.729455 17276 ts_tablet_manager.cc:1367] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:57:52.729871 17276 raft_consensus.cc:348] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.729970 17276 raft_consensus.cc:374] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.729998 17276 raft_consensus.cc:729] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.730144 17276 consensus_queue.cc:260] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.730618 17278 ts_tablet_manager.cc:1888] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240501 19:57:52.730695 17278 log.cc:1196] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Deleting WAL directory at /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/ts-1/wal/wals/d42e1516bbc24af8b461d97ac766d28a
I20240501 19:57:52.731149 17032 raft_consensus.cc:482] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:52.731237 17032 raft_consensus.cc:504] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.731503 17276 ts_tablet_manager.cc:1398] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:52.731650 17032 leader_election.cc:288] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:52.731669 17276 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:52.732370 17280 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:52.732466 17280 ts_tablet_manager.cc:1367] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240501 19:57:52.732685 17276 tablet_bootstrap.cc:654] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.733028 17280 raft_consensus.cc:348] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.733130 17280 raft_consensus.cc:374] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.733161 17280 raft_consensus.cc:729] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.733292 17280 consensus_queue.cc:260] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.733628 17280 ts_tablet_manager.cc:1398] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:52.733819 17280 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:52.733939 17279 tablet_bootstrap.cc:492] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:52.734022 17279 ts_tablet_manager.cc:1367] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 19:57:52.734146 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccb5505f0db4729abf237bc5e6fe492" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:52.734272 15326 raft_consensus.cc:2454] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:52.734531 17279 raft_consensus.cc:348] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.734653 17279 raft_consensus.cc:374] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.734694 17279 raft_consensus.cc:729] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.734714 17280 tablet_bootstrap.cc:654] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.734839 17279 consensus_queue.cc:260] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.734871 15397 leader_election.cc:302] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:52.735139 17279 ts_tablet_manager.cc:1398] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:52.735312 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccb5505f0db4729abf237bc5e6fe492" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:52.735425 15599 raft_consensus.cc:2454] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 0.
I20240501 19:57:52.735423 17032 raft_consensus.cc:2781] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:52.735476 17032 raft_consensus.cc:482] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:52.735513 17032 raft_consensus.cc:3037] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.735316 17279 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:52.736582 17032 raft_consensus.cc:504] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.737571 17279 tablet_bootstrap.cc:654] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.738560 17032 leader_election.cc:288] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Requested vote from peers 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023), 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:52.738675 17278 ts_tablet_manager.cc:1909] T d42e1516bbc24af8b461d97ac766d28a P 68539c0ecd3e44e080cb53987d4f4c23: Deleting consensus metadata
I20240501 19:57:52.739244 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccb5505f0db4729abf237bc5e6fe492" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:52.739368 15599 raft_consensus.cc:3037] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.739634 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ccb5505f0db4729abf237bc5e6fe492" candidate_uuid: "68539c0ecd3e44e080cb53987d4f4c23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:52.739750 15326 raft_consensus.cc:3037] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.740134 17280 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:52.740217 17280 ts_tablet_manager.cc:1367] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:57:52.740435 15599 raft_consensus.cc:2454] T 7ccb5505f0db4729abf237bc5e6fe492 P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:52.740701 17280 raft_consensus.cc:348] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.740788 17280 raft_consensus.cc:374] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.740809 17280 raft_consensus.cc:729] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.740784 15326 raft_consensus.cc:2454] T 7ccb5505f0db4729abf237bc5e6fe492 P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68539c0ecd3e44e080cb53987d4f4c23 in term 1.
I20240501 19:57:52.741009 17280 consensus_queue.cc:260] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.741211 15397 leader_election.cc:302] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
I20240501 19:57:52.741309 17280 ts_tablet_manager.cc:1398] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:52.741382 17032 raft_consensus.cc:2781] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:52.741474 17032 raft_consensus.cc:686] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 LEADER]: Becoming Leader. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Running, Role: LEADER
I20240501 19:57:52.741492 17280 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23: Bootstrap starting.
I20240501 19:57:52.741601 17032 consensus_queue.cc:237] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } } peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } }
I20240501 19:57:52.742364 17280 tablet_bootstrap.cc:654] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.744302 17279 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:52.744393 17279 ts_tablet_manager.cc:1367] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:52.744938 17279 raft_consensus.cc:348] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.745046 17279 raft_consensus.cc:374] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.745079 17279 raft_consensus.cc:729] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.745210 17276 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:52.745203 17279 consensus_queue.cc:260] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.745301 17276 ts_tablet_manager.cc:1367] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.002s
I20240501 19:57:52.745514 17279 ts_tablet_manager.cc:1398] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:52.745684 17279 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:52.745790 17276 raft_consensus.cc:348] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.745884 17276 raft_consensus.cc:374] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.745913 17276 raft_consensus.cc:729] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.746063 17276 consensus_queue.cc:260] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.746687 17279 tablet_bootstrap.cc:654] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.747592 17276 ts_tablet_manager.cc:1398] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:52.747786 17276 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d: Bootstrap starting.
I20240501 19:57:52.748631 17276 tablet_bootstrap.cc:654] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.749644 17280 tablet_bootstrap.cc:492] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23: No bootstrap required, opened a new log
I20240501 19:57:52.749727 17280 ts_tablet_manager.cc:1367] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:57:52.750294 17280 raft_consensus.cc:348] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.750403 17280 raft_consensus.cc:374] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.750437 17280 raft_consensus.cc:729] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68539c0ecd3e44e080cb53987d4f4c23, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.750500 17279 tablet_bootstrap.cc:492] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:52.750581 17279 ts_tablet_manager.cc:1367] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:52.750561 17280 consensus_queue.cc:260] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.750873 17280 ts_tablet_manager.cc:1398] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:57:52.751029 17279 raft_consensus.cc:348] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.751125 17279 raft_consensus.cc:374] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.751156 17279 raft_consensus.cc:729] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.751274 17279 consensus_queue.cc:260] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.751542 17279 ts_tablet_manager.cc:1398] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:52.751709 17279 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432: Bootstrap starting.
I20240501 19:57:52.752645 17279 tablet_bootstrap.cc:654] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:52.753822 17276 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d: No bootstrap required, opened a new log
I20240501 19:57:52.753906 17276 ts_tablet_manager.cc:1367] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:57:52.754381 17276 raft_consensus.cc:348] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.754504 17276 raft_consensus.cc:374] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.754534 17276 raft_consensus.cc:729] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.754652 17276 consensus_queue.cc:260] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.754928 17276 ts_tablet_manager.cc:1398] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:52.755614 17279 tablet_bootstrap.cc:492] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432: No bootstrap required, opened a new log
I20240501 19:57:52.755692 17279 ts_tablet_manager.cc:1367] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:52.756132 17279 raft_consensus.cc:348] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.756227 17279 raft_consensus.cc:374] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:52.756258 17279 raft_consensus.cc:729] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Initialized, Role: FOLLOWER
I20240501 19:57:52.756368 17279 consensus_queue.cc:260] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.756645 17279 ts_tablet_manager.cc:1398] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:52.847301 16937 raft_consensus.cc:482] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:52.847491 16937 raft_consensus.cc:504] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.848052 16937 leader_election.cc:288] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:52.848539 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eef7b37f4d46818c14c0caf8c671ee" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d" is_pre_election: true
I20240501 19:57:52.848539 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eef7b37f4d46818c14c0caf8c671ee" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:52.848706 15463 raft_consensus.cc:2454] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:52.848706 15326 raft_consensus.cc:2454] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 0.
I20240501 19:57:52.849057 15533 leader_election.cc:302] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:52.849256 16937 raft_consensus.cc:2781] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:52.849321 16937 raft_consensus.cc:482] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:52.849356 16937 raft_consensus.cc:3037] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.850229 16937 raft_consensus.cc:504] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.850626 16937 leader_election.cc:288] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247), 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:52.851006 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eef7b37f4d46818c14c0caf8c671ee" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:52.851019 15326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eef7b37f4d46818c14c0caf8c671ee" candidate_uuid: "7f011d422e164c949a9d54d846b7c432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8dfc11dc7542f583644b9ac747545d"
I20240501 19:57:52.851145 15463 raft_consensus.cc:3037] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.851176 15326 raft_consensus.cc:3037] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.851922 15463 raft_consensus.cc:2454] T 25eef7b37f4d46818c14c0caf8c671ee P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:52.852221 15326 raft_consensus.cc:2454] T 25eef7b37f4d46818c14c0caf8c671ee P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f011d422e164c949a9d54d846b7c432 in term 1.
I20240501 19:57:52.852229 15536 leader_election.cc:302] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68539c0ecd3e44e080cb53987d4f4c23, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:52.852423 16937 raft_consensus.cc:2781] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:52.852622 16937 raft_consensus.cc:686] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [term 1 LEADER]: Becoming Leader. State: Replica: 7f011d422e164c949a9d54d846b7c432, State: Running, Role: LEADER
I20240501 19:57:52.852792 16937 consensus_queue.cc:237] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.894754 17013 raft_consensus.cc:482] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:52.894901 17013 raft_consensus.cc:504] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.895543 17013 leader_election.cc:288] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:52.895928 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5bc7534f934590805794409e81438e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:52.895902 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5bc7534f934590805794409e81438e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:52.896064 15599 raft_consensus.cc:2454] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:52.896063 15463 raft_consensus.cc:2454] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:52.896395 15261 leader_election.cc:302] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:52.896600 17013 raft_consensus.cc:2781] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:52.896667 17013 raft_consensus.cc:482] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:52.896703 17013 raft_consensus.cc:3037] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.897629 17013 raft_consensus.cc:504] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.897990 17013 leader_election.cc:288] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
W20240501 19:57:52.898415 15261 connection.cc:535] server connection from 127.0.0.1:42578 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:52.898404 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5bc7534f934590805794409e81438e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:52.898411 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5bc7534f934590805794409e81438e" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:52.898522 15599 raft_consensus.cc:3037] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.898522 15463 raft_consensus.cc:3037] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:57:52.899449 15398 connection.cc:535] client connection to 127.14.96.60:40075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:52.899614 15599 raft_consensus.cc:2454] T 2b5bc7534f934590805794409e81438e P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
W20240501 19:57:52.899623 15398 connection.cc:535] server connection from 127.0.0.1:34180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:52.899613 15463 raft_consensus.cc:2454] T 2b5bc7534f934590805794409e81438e P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
W20240501 19:57:52.899936 15260 connection.cc:535] client connection to 127.14.96.60:40075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:52.900151 15654 connection.cc:535] client connection to 127.14.96.60:40075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:52.900475 15262 leader_election.cc:302] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 68539c0ecd3e44e080cb53987d4f4c23; no voters:
W20240501 19:57:52.900601 15654 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.14.96.60:40075)
I20240501 19:57:52.900652 17013 raft_consensus.cc:2781] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:52.900732 17013 raft_consensus.cc:686] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:52.900902 17013 consensus_queue.cc:237] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
W20240501 19:57:52.901008 15654 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:52.901412 15654 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:52.901521 15654 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.14.96.60:40075)
W20240501 19:57:52.903875 15534 connection.cc:535] client connection to 127.14.96.60:40075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:57:52.904047 15534 connection.cc:535] server connection from 127.0.0.1:58962 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:57:52.934953 17305 raft_consensus.cc:482] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:52.935106 17305 raft_consensus.cc:504] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.935550 17305 leader_election.cc:288] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:52.935995 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a2213e8f4f16bf77a9dc19c3cbab" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432" is_pre_election: true
I20240501 19:57:52.935993 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a2213e8f4f16bf77a9dc19c3cbab" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23" is_pre_election: true
I20240501 19:57:52.936137 15599 raft_consensus.cc:2454] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:52.936167 15463 raft_consensus.cc:2454] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 0.
I20240501 19:57:52.936445 15261 leader_election.cc:302] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:52.936625 17305 raft_consensus.cc:2781] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:52.936686 17305 raft_consensus.cc:482] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:52.936728 17305 raft_consensus.cc:3037] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.937645 17305 raft_consensus.cc:504] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:52.938010 17305 leader_election.cc:288] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Requested vote from peers 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181), 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:52.938453 15599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a2213e8f4f16bf77a9dc19c3cbab" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f011d422e164c949a9d54d846b7c432"
I20240501 19:57:52.938584 15599 raft_consensus.cc:3037] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.938555 15463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a8a2213e8f4f16bf77a9dc19c3cbab" candidate_uuid: "0c8dfc11dc7542f583644b9ac747545d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68539c0ecd3e44e080cb53987d4f4c23"
I20240501 19:57:52.938663 15463 raft_consensus.cc:3037] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:52.939610 15599 raft_consensus.cc:2454] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 7f011d422e164c949a9d54d846b7c432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:52.939610 15463 raft_consensus.cc:2454] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 68539c0ecd3e44e080cb53987d4f4c23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8dfc11dc7542f583644b9ac747545d in term 1.
I20240501 19:57:52.939918 15261 leader_election.cc:302] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8dfc11dc7542f583644b9ac747545d, 7f011d422e164c949a9d54d846b7c432; no voters:
I20240501 19:57:52.940150 17305 raft_consensus.cc:2781] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:52.940245 17305 raft_consensus.cc:686] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [term 1 LEADER]: Becoming Leader. State: Replica: 0c8dfc11dc7542f583644b9ac747545d, State: Running, Role: LEADER
I20240501 19:57:52.940398 17305 consensus_queue.cc:237] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 } } peers { permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 } } peers { permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 } }
I20240501 19:57:53.188606 17259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 62106a6dbb3846df9d3b29ab99fb3763)
I20240501 19:57:53.188757 17259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:53.189281 17259 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:53.190379 17154 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:53.190467 17153 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:53.190534 17153 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:53.190694 17259 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:57:53.211037 17032 consensus_queue.cc:1035] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.214536 17032 consensus_queue.cc:1035] T 7ccb5505f0db4729abf237bc5e6fe492 P 68539c0ecd3e44e080cb53987d4f4c23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.270012 17303 consensus_queue.cc:1035] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" member_type: VOTER last_known_addr { host: "127.14.96.1" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.273844 16937 consensus_queue.cc:1035] T 25eef7b37f4d46818c14c0caf8c671ee P 7f011d422e164c949a9d54d846b7c432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.286415 17305 consensus_queue.cc:1035] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.290890 17013 consensus_queue.cc:1035] T 2b5bc7534f934590805794409e81438e P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.462642 14720 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu with pid 17131
I20240501 19:57:53.537771 17322 consensus_queue.cc:1035] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f011d422e164c949a9d54d846b7c432" member_type: VOTER last_known_addr { host: "127.14.96.3" port: 42023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:53.541199 17013 consensus_queue.cc:1035] T d3a8a2213e8f4f16bf77a9dc19c3cbab P 0c8dfc11dc7542f583644b9ac747545d [LEADER]: Connected to new peer: Peer: permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" member_type: VOTER last_known_addr { host: "127.14.96.2" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:54.336273 14720 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
/tmp/dist-test-tasknpHgT3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--webserver_interface=127.14.96.61
--webserver_port=42603
--builtin_ntp_servers=127.14.96.20:42869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240501 19:57:54.453011 17337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:57:54.453313 17337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:57:54.453379 17337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:57:54.456907 17337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:57:54.456971 17337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:57:54.456992 17337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:57:54.457011 17337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:57:54.461023 17337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.14.96.20:42869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:40981
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.14.96.61:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.14.96.61
--webserver_port=42603
--never_fsync=true
--heap_profile_path=/tmp/kudu.17337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:57:54.462184 17337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:57:54.463339 17337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:54.471467 17347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:54.472088 17346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:54.472693 17349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:54.473431 17337 server_base.cc:1034] running on GCE node
I20240501 19:57:54.473949 17337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:57:54.474926 17337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:57:54.476092 17337 hybrid_clock.cc:648] HybridClock initialized: now 1714593474476075 us; error 32 us; skew 500 ppm
I20240501 19:57:54.478194 17337 webserver.cc:460] Webserver started at http://127.14.96.61:42603/ using document root <none> and password file <none>
I20240501 19:57:54.478761 17337 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:54.478819 17337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:54.482267 17337 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:57:54.484392 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:54.485523 17337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:57:54.485631 17337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data,/tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
uuid: "72d8e36923de4d1e8a6de93e73c84658"
format_stamp: "Formatted at 2024-05-01 19:57:24 on dist-test-slave-spj6"
I20240501 19:57:54.486054 17337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:54.499642 17337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:54.500312 17337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:57:54.500473 17337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:54.507228 17337 rpc_server.cc:307] RPC server started. Bound to: 127.14.96.61:46321
I20240501 19:57:54.507259 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.14.96.61:46321 every 8 connection(s)
I20240501 19:57:54.508379 17337 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknpHgT3/test-tmp/master-stress-test.2.CatalogManagerConfigurations_MasterStressTest.Test_2.1714593429527495-14720-0/minicluster-data/master-1/data/info.pb
I20240501 19:57:54.512683 14720 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknpHgT3/build/debug/bin/kudu as pid 17337
I20240501 19:57:54.513123 17408 sys_catalog.cc:263] Verifying existing consensus state
I20240501 19:57:54.515669 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap starting.
I20240501 19:57:54.530565 17408 log.cc:824] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Log is configured to *not* fsync() on all Append() calls
I20240501 19:57:54.612746 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap replayed 1/1 log segments. Stats: ops{read=168 overwritten=0 applied=168 ignored=0} inserts{seen=111 ignored=0} mutations{seen=147 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:54.613252 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: Bootstrap complete.
I20240501 19:57:54.616386 17408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:54.616734 17408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d8e36923de4d1e8a6de93e73c84658, State: Initialized, Role: FOLLOWER
I20240501 19:57:54.617214 17408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 168, Last appended: 8.168, Last appended by leader: 168, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:54.617684 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 8 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } } }
I20240501 19:57:54.617808 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:57:54.618312 17408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:54.623607 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:57:54.625367 17427 catalog_manager.cc:1259] Loaded cluster ID: 80d63e91387f4814881515380f6e255e
I20240501 19:57:54.625425 17427 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: loading cluster ID for follower catalog manager: success
I20240501 19:57:54.625558 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:54.629797 17427 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: acquiring CA information for follower catalog manager: success
I20240501 19:57:54.633100 17427 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:57:54.638690 15374 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:54.641096 17372 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0c8dfc11dc7542f583644b9ac747545d" instance_seqno: 1714593445669439) as {username='slave'} at 127.14.96.1:51741; Asking this server to re-register.
I20240501 19:57:54.641960 15374 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:54.643961 17371 ts_manager.cc:194] Registered new tserver with Master: 0c8dfc11dc7542f583644b9ac747545d (127.14.96.1:36247)
I20240501 19:57:54.837863 15512 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:54.839205 17371 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "68539c0ecd3e44e080cb53987d4f4c23" instance_seqno: 1714593445869000) as {username='slave'} at 127.14.96.2:47899; Asking this server to re-register.
I20240501 19:57:54.839668 15512 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:54.840700 17371 ts_manager.cc:194] Registered new tserver with Master: 68539c0ecd3e44e080cb53987d4f4c23 (127.14.96.2:34181)
I20240501 19:57:54.880506 15648 heartbeater.cc:347] Connected to a master server at 127.14.96.61:46321
I20240501 19:57:54.881922 17371 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "7f011d422e164c949a9d54d846b7c432" instance_seqno: 1714593446063103) as {username='slave'} at 127.14.96.3:60913; Asking this server to re-register.
I20240501 19:57:54.882442 15648 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:54.883545 17371 ts_manager.cc:194] Registered new tserver with Master: 7f011d422e164c949a9d54d846b7c432 (127.14.96.3:42023)
I20240501 19:57:55.082672 17413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:55.082911 17413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:55.084115 17413 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:55.084605 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:55.085435 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:55.085554 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:55.085649 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:55.085884 17413 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:57:55.759429 17445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:55.759580 17445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:55.760047 17445 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:55.761605 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:55.761705 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:55.761795 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:55.761947 17445 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:57:56.360114 17457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:56.360253 17457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:56.360687 17457 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:56.361919 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:56.362021 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:56.362111 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:56.362263 17457 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:57:56.654109 14720 master-stress-test.cc:578] Killing test thread 1/25
I20240501 19:57:57.739693 17477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:57.739859 17477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:57.740355 17477 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:57.741660 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:57:57.741817 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:57:57.741899 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:57.742074 17477 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:57:59.626557 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:57:59.626679 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:57:59.826512 17503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:59.826653 17503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:57:59.827143 17503 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:57:59.828310 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:57:59.828419 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:57:59.828517 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:57:59.828663 17503 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:03.231101 17551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:03.231264 17551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:03.231652 17551 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:03.232206 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:03.233063 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:03.233161 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:03.233268 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:03.233444 17551 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:04.627806 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:04.627933 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:07.927935 17618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:07.928066 17618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:07.928485 17618 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:07.929663 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:07.929801 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:07.929879 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:07.930025 17618 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:09.629065 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:09.629213 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:11.812708 17674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:11.812883 17674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:11.813364 17674 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:11.813933 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 19:58:11.814652 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:11.814749 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:11.814836 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:11.814997 17674 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:14.630234 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:14.630369 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:19.631317 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:19.631440 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:22.029479 17818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:22.029618 17818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:22.030021 17818 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:22.030573 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:58:22.031208 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:58:22.031287 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:58:22.031373 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:22.031526 17818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:24.082947 17849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:24.083114 17849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:24.083576 17849 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:24.084728 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:24.084816 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:24.084920 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:24.085047 17849 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:24.632508 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:24.632624 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 19:58:26.199666 15373 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (96 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:26.407948 15510 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (75 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:58:26.941519 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (76 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:29.633785 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:29.633908 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:30.042516 17937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:30.042747 17937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:30.043247 17937 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:30.043783 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 19:58:30.044523 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:58:30.044618 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:58:30.044703 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:30.044888 17937 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:32.009532 17970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:32.009681 17970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:32.010116 17970 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:32.011320 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:58:32.011412 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:58:32.011503 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:32.011641 17970 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:34.634979 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:34.635136 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:36.477516 18037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:36.477650 18037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:36.478057 18037 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:36.478595 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 19:58:36.479303 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:36.479434 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:36.479516 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:36.479671 18037 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:39.636179 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:39.636296 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:44.637279 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:44.637403 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:49.638693 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:49.638823 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:54.639940 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:54.640097 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:58:56.082815 18309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:56.082959 18309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:56.083756 18309 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:56.084143 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:58:56.084884 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:56.085021 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:56.085078 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:56.085214 18309 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:57.814756 18335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:58:57.814903 18335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:58:57.815330 18335 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:58:57.816581 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:58:57.816673 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:58:57.816751 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:58:57.816928 18335 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:58:59.641345 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:58:59.641502 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:04.642557 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:04.642688 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:05.107987 18442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:05.108124 18442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:59:05.108516 18442 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:59:05.109063 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 19:59:05.109769 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:05.109923 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:59:05.110030 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:59:05.110163 18442 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:59:09.643795 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:09.643926 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:12.681897 18547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:12.682037 18547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:59:12.682451 18547 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:59:12.682991 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:59:12.683784 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:12.683892 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:59:12.683955 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:59:12.684094 18547 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:59:13.758559 18565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:13.758692 18565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:59:13.759073 18565 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:59:13.760197 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:13.760284 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:59:13.760375 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:59:13.760509 18565 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:59:14.644968 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:14.645124 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:19.650104 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:19.650244 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:24.651372 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:24.651515 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 19:59:26.324507 15373 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (216 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:26.528790 15511 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (211 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:27.066253 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (196 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:59:29.652639 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:29.652767 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:32.551111 18829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:32.551272 18829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:59:32.551849 18829 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:59:32.552455 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 19:59:32.553284 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 19:59:32.553370 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 19:59:32.553460 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:59:32.553602 18829 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:59:34.654081 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:34.654229 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:39.655380 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:39.655521 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:44.656603 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:44.656737 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:49.443320 19068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:59:49.443493 19068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 19:59:49.444098 19068 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 19:59:49.444757 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 19:59:49.445468 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 19:59:49.445562 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 19:59:49.445641 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 19:59:49.445797 19068 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 19:59:49.657636 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:49.657768 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:54.659030 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:54.659152 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 19:59:54.663980 17361 maintenance_manager.cc:419] P 72d8e36923de4d1e8a6de93e73c84658: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033346
I20240501 19:59:54.709813 17360 maintenance_manager.cc:643] P 72d8e36923de4d1e8a6de93e73c84658: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.019s sys 0.004s Metrics: {"bytes_written":31688,"cfile_init":1,"dirs.queue_time_us":236,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":1191,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":586,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":85,"thread_start_us":205,"threads_started":2,"wal-append.queue_time_us":249}
I20240501 19:59:54.710844 17361 maintenance_manager.cc:419] P 72d8e36923de4d1e8a6de93e73c84658: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6529 bytes on disk
I20240501 19:59:54.711536 17360 maintenance_manager.cc:643] P 72d8e36923de4d1e8a6de93e73c84658: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 19:59:59.660178 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 19:59:59.660302 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:00.063156 19213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:00.063308 19213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:00.063727 19213 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:00.064179 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:00:00.064944 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:00.065037 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:00.065119 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:00.065287 19213 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:04.661491 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:04.661620 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:09.662806 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:09.662950 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:11.132867 19412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:11.133051 19412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:11.133566 19412 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:11.134392 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:00:11.135138 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:11.135231 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:11.135296 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:11.135496 19412 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:14.664222 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:14.664392 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:19.665602 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:19.665736 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:23.918756 19638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:23.918911 19638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:23.919525 19638 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:23.920189 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:00:23.920909 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:23.921083 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:23.921161 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:23.921293 19638 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:24.666810 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:24.666980 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:00:26.449987 15373 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (336 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:26.655782 15511 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (331 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:27.195147 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (316 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:29.668051 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:29.668212 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:34.053354 19800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:34.053495 19800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:34.053859 19800 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:34.054482 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:00:34.055230 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:00:34.055321 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:00:34.055402 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:34.055541 19800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:34.669268 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:34.669412 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:37.233971 19857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:37.234124 19857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:37.234616 19857 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:37.235891 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:00:37.236227 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:00:37.236320 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:37.236497 19857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:39.671365 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:39.671492 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:44.672691 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:44.672830 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:44.739344 19987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:44.739476 19987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:44.739993 19987 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:44.740386 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:00:44.741181 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:00:44.741369 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:00:44.741451 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:44.741592 19987 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:49.673988 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:49.674124 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:51.586694 20102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:51.586841 20102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:51.587253 20102 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:51.587812 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:00:51.588516 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:00:51.588737 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:00:51.588793 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:51.588960 20102 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:53.449275 20135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:53.449414 20135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:53.449846 20135 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:53.451170 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:53.451278 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:53.451341 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:53.451475 20135 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:54.353701 20156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:00:54.353833 20156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:00:54.354307 20156 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:00:54.355618 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:00:54.355723 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:00:54.355794 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:00:54.355958 20156 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:00:54.675163 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:54.675287 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:00:59.676471 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:00:59.676597 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:04.677831 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:04.677983 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:06.039677 20357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.039820 20357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:06.040261 20357 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:06.040860 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240501 20:01:06.041939 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:01:06.042047 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:01:06.042109 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:06.042376 20357 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:06.822294 20373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:06.822438 20373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:06.822835 20373 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:06.824064 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:01:06.824174 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:01:06.824258 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:06.824401 20373 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:09.679289 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:09.679472 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:14.680716 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:14.680888 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:19.682169 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:19.682319 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:23.115707 20649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:23.115854 20649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:23.116428 20649 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:23.117065 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:01:23.117707 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:01:23.117825 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:01:23.117902 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:23.118050 20649 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:24.683483 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:24.683622 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:01:26.575605 15373 heartbeater.cc:644] Failed to heartbeat to 127.14.96.62:37747 (456 consecutive failures): Network error: Failed to ping master at 127.14.96.62:37747: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:01:26.777102 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:26.777238 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:26.777702 20713 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:26.779148 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:01:26.779248 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:01:26.779335 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
W20240501 20:01:26.779422 15510 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (435 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:01:26.779534 20713 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240501 20:01:27.320112 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (436 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:01:29.684726 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:29.684898 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:34.686214 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:34.686347 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:36.246482 20872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:36.246749 20872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:36.247418 20872 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:36.248239 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:01:36.249110 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:01:36.249207 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:01:36.249291 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:36.249436 20872 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:39.687569 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:39.687702 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:44.688863 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:44.689024 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:45.462531 21030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:45.462689 21030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:45.463193 21030 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:45.463963 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:01:45.464694 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:01:45.464798 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:01:45.464939 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:45.465122 21030 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:49.473503 21102 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:01:49.473681 21102 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:01:49.474186 21102 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:01:49.475631 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:01:49.475732 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:01:49.475843 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:01:49.476009 21102 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:01:49.690057 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:49.690223 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:54.691239 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:54.691390 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:01:59.692555 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:01:59.692687 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:01.436687 21303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:01.436887 21303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:01.437304 21303 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:01.437862 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:02:01.438725 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:01.438851 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:01.438936 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:01.439081 21303 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:04.693723 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:04.693881 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:09.181069 21434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:09.181214 21434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:09.181623 21434 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:09.182168 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:02:09.182822 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:09.183004 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:09.183074 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:09.183216 21434 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:09.697345 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:09.697482 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:14.698558 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:14.698698 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:16.527297 21555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:16.527426 21555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:16.527850 21555 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:16.528398 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:02:16.529105 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:02:16.529213 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:16.529294 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:16.529440 21555 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:19.699656 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:19.699811 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:21.876691 21646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:21.876912 21646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:21.877349 21646 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:21.877905 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:02:21.878548 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:21.878674 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:21.878726 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:21.878854 21646 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:22.428187 21662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:22.428329 21662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:22.428970 21662 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:22.430209 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:22.430313 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:22.430375 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:22.430526 21662 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:24.701031 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:24.701174 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240501 20:02:26.698532 15372 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (556 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:26.902410 15510 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (555 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:27.102418 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:27.102566 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:27.102988 21741 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:27.103636 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:02:27.104377 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:27.104660 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:27.104737 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:27.105028 21741 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240501 20:02:27.444751 15646 heartbeater.cc:644] Failed to heartbeat to 127.14.96.60:40075 (556 consecutive failures): Network error: Failed to ping master at 127.14.96.60:40075: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:29.702448 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:29.702594 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:34.703794 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:34.703931 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:37.344377 21907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:37.344518 21907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:37.345021 21907 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:37.345551 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:02:37.346200 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:02:37.346331 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:37.346414 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:37.346555 21907 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:39.190476 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:39.190608 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:39.191051 21943 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:39.192265 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:02:39.192363 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:39.192430 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:39.192565 21943 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:39.705046 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:39.705178 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:44.706323 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:44.706462 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:45.154035 22046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:45.154189 22046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:45.154649 22046 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:45.155118 17358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:02:45.155835 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
W20240501 20:02:45.155915 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
I20240501 20:02:45.155997 17357 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:45.156155 22046 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:49.707685 17337 master_runner.cc:186] Error getting master registration for 127.14.96.62:37747: Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111),
I20240501 20:02:49.707860 17337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240501 20:02:52.317874 22160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:02:52.318038 22160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f177158f524f0183f6eec721ea83e4" member_type: VOTER last_known_addr { host: "127.14.96.62" port: 37747 } } peers { permanent_uuid: "72d8e36923de4d1e8a6de93e73c84658" member_type: VOTER last_known_addr { host: "127.14.96.61" port: 46321 } } peers { permanent_uuid: "62106a6dbb3846df9d3b29ab99fb3763" member_type: VOTER last_known_addr { host: "127.14.96.60" port: 40075 } }
I20240501 20:02:52.318526 22160 leader_election.cc:288] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747), 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075)
W20240501 20:02:52.319142 17357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:02:52.319916 17357 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 62106a6dbb3846df9d3b29ab99fb3763 (127.14.96.60:40075): Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:52.319999 17358 leader_election.cc:334] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer e9f177158f524f0183f6eec721ea83e4 (127.14.96.62:37747): Network error: Client connection negotiation failed: client connection to 127.14.96.62:37747: connect: Connection refused (error 111)
I20240501 20:02:52.320067 17358 leader_election.cc:302] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72d8e36923de4d1e8a6de93e73c84658; no voters: 62106a6dbb3846df9d3b29ab99fb3763, e9f177158f524f0183f6eec721ea83e4
I20240501 20:02:52.320225 22160 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72d8e36923de4d1e8a6de93e73c84658 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240501 20:02:52.720317 17361 maintenance_manager.cc:419] P 72d8e36923de4d1e8a6de93e73c84658: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 6529 bytes on disk
I20240501 20:02:52.723363 17360 maintenance_manager.cc:643] P 72d8e36923de4d1e8a6de93e73c84658: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.003s sys 0.000s Metrics: {"dirs.queue_time_us":210,"thread_start_us":257,"threads_started":2,"wal-append.queue_time_us":232}
W20240501 20:02:53.586977 15653 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:53.587204 15653 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:53.587286 15653 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
W20240501 20:02:53.587323 15653 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15684 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Aborted at 1714593773 (unix time) try "date -d @1714593773" if you are using GNU date ***
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800003980) received by PID 14720 (TID 0x7f8b8280a700) from PID 14720; stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f8b9c8fd980 (unknown) at ??:0
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F2F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605999 15675 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605999 15675 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606191 15662 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_206463f213f945b2a74cbbcddbada5e8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 1 @ 0x7f8b9a54ffb7 gsignal
11)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605999 15675 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606191 15662 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_206463f213f945b2a74cbbcddbada5e8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606001 15680 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605999 15675 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606191 15662 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_206463f213f945b2a74cbbcddbada5e8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606001 15680 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.607189 15664 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20240501 20:02:Timed out: Error creating table default.Table_a62bb6dcb7644aa4bb9008b9f9a196dd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)53.607170 15682 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
F20240501 20:02:53.598299 15673 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.598348 15660 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_abc8d72a6cc74062a62861c18c185a62 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15668 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_7a325fdcd5224571b886231cf98de803 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599561 15671 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.599494 15669 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_84bf2da3cdaf45fa84c6099c013783e6 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600736 15681 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.600775 15677 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.601989 15667 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_189735560cf84873a69187bf084e7654 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602022 15666 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_06ea968fee694fad8d382f41b03ea362 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602110 15685 master-stress-test.cc:392] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602139 15678 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.602288 15674 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.603317 15661 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a283fbed2b5748f1b138b73c95bb2dc3 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604458 15676 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.604530 15665 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_f18a22bd8291471ba13555510194e198 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605777 15683 master-stress-test.cc:329] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605911 15670 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_a82d93e7c3e949e99304ee3a61f2f5e9 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605785 15672 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606047 15679 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.605999 15675 master-stress-test.cc:352] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606191 15662 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_206463f213f945b2a74cbbcddbada5e8 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.606001 15680 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: DeleteTable timed out after deadline expired: DeleteTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)F20240501 20:02:53.607189 15664 master-stress-test.cc:298] Check failed: _s.ok() Bad status: F20240501 20:02:Timed out: Error creating table default.Table_a62bb6dcb7644aa4bb9008b9f9a196dd on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)53.607170 15682 master-stress-test.cc:298] Check failed: _s.ok() Bad status: Timed out: ReplaceTablet timed out after deadline expired: ReplaceTablet passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.14.96.62:37747,127.14.96.61:46321,127.14.96.60:40075, num_attempts: 772) passed its deadline: Network error: Client connection negotiation failed: client connection to 127.14.96.60:40075: connect: Connection refused (error 111)
*** Check failure stack trace: ***
@ 0x7f8b9a551921 abort at ??:0
@ 0x7f8b9b3bcdcd google::LogMessage::Fail() at ??:0
@ 0x7f8b9b3c0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8b9b3bc7cc google::LogMessage::Flush() at ??:0
@ 0x7f8b9b3bdf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x56409a8d8bc5 kudu::MasterStressTest::AlterTableThread() at ??:0
@ 0x56409a8eb76a std::__invoke_impl<>() at ??:0
@ 0x56409a8e9400 std::__invoke<>() at ??:0
@ 0x56409a8f0a8f _ZNSt6thread8_InvokerISt5tupleIJMN4kudu16MasterStressTestEFvvEPNS2_26MasterStressTest_Test_TestEEEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE at ??:0
@ 0x56409a8efb05 std::thread::_Invoker<>::operator()() at ??:0
@ 0x56409a8ee92a std::thread::_State_impl<>::_M_run() at ??:0
@ 0x7f8b9abd96df (unknown) at ??:0
@ 0x7f8b9c8f26db start_thread at ??:0
@ 0x7f8b9a63271f clone at ??:0